A0-A161  649 
UNCLASSIFIED 


EVALUATION  OF  PROCEDURES  EHPLOVED  DURING  SOURCE  1/2 

SELECTION  FOR  CONTRACTS  I.  .  <U)  AIR  FORCE  INST  OF  TECH 
NRIGHT-PATTERSON  AFB  OH  SCHOOL  OF  SVST. .  J  E  PUGH 
SEP  85  RF I T/GSH/LSV/85S-28  FFG  5/1  NL 


AD-A161  649 


The  contents  of  the  document  are  technically  accurate,  and 
no  sensitive  items,  detrimental  ideas,  or  deleterious 
information  are  contained  therein.  Furthermore,  the  views 
expressed  in  the  document  are  those  of  the  author (s)  and  do 
not  necessarily  reflect  the  views  of  the  School  of  Systems 
and  Logistics,  the  Air  University,  the  United  States  Air 
Force,  or  the  Department  of  Defense. 


AF I T /6SM/LSY /85S-30 


EVALUATION  OF  PROCEDURES  EMPLOYED  DURING  SOURCE 
SELECTION  FOR  CONTRACTS  INCLUDING  CLAUSES  REQUIRING 
COST/SCHEDULE  CONTROL  SYSTEMS  CRITERIA  (C/SCSC) 


THESIS 


Presented  to  the  Faculty  of  the 
School  of  Systems  and  Logistics 
of  the  Air  Force  Institute  of  Technology 
Air  University 

In  Partial  Fulfillment  of  the 
Requirements  for  the  Degree  of 
Master  of  Science  in  Systems  Management 


James  E.  Pugh,  B.S. 
Captain,  USAF 

September  1985 


Approved  for  public  release:  distribution  unlimited 


Acknowl edgement s 


I  extend  my  gratitude  to  all  who  helped  to  make  this 
research  possible.  I  am  deeply  indebted  to  my  -faculty 
advisor,  Lt  Col  Thomas  L.  Bowman,  for  his  continuing 
patience  and  assistance  in  times  of  need.  I  also  wish  to 
thank  my  reader,  Lt  Col  Jeffery  Phillips.  Finally,  I  wish 
to  thank  all  those  at  ASD  who  took  time  out  of  their 
schedules  to  be  interviewed. 

James  E.  Pugh 


Acknowledgements  ......  .  . 

List  of  Figures  . 

Abstract . . . 

I.  Introduction  ............. 

General  Issue  .....  . 

Specific  Problem  .....  . 

Definitions . . . 

Checklist  . 

Work  Breakdown  Structure  <WBS) 
Source  Selection  ........ 

Research  Objective  . 

Research  Questions  ..  . 

II.  Literature  Review  ........... 

Topic  Statement  ...  . 

Justification  .....  . 

Scope . . . 

Discussion . . 

Purposes  and  Objectives  of  C/SCSC 
Criteria  and  the 

Implementation  Process  . 

Problems  and  Perceptions  of  C/SCSC 
Results  of  Literature  Review  .... 

III.  Methodology  . . . 

Initial  Research  Phase  ....... 

Primary  Research  Phase  . 

Reasons  for  Methodology  Used  .... 

IV.  Findings  .........  . 

Research  Question  1  ........ 

Research  Question  2  .  . 

The  Source  Selection  Elements 

The  Evaluation  Process  . 

Research  Question  3  ...  . 

Research  Question  4  . 

Preparing  for  the  Evaluation  .  . 
Limitations  and  Problems 

with  the  Evaluation  . 

The  Evaluation  . 

Summary  of  Research  Findings  .... 


page 


V.  Conclusions  .  49 

C/SC5C  Items  to  Consider  during 

Source  Selection  .  51 

Recommendation  . .  52 

Appendix  A.  Evaluation/Demonstration  Review 

Checklist  For  C/SCSC .  54 

Appendix  Bs  Record  o-f  Interviews  .........  67 

Appendix  C:  ASD  Evaluation  Standards  .  81 

Appendix  D:  Evaluation  Write-ups  .........  84 

Bibliography  .................  91 

Vita .  94 


Figure 

1.  Source  Selection  Hierarchy  . 

2.  Source  Selection  Schedule  . 

3.  C/SCSC  In  Relation  To  Overall  Program 


AF I T /GSM/LSY /85S-30 

Abstract 

To  effectively  manage  Research  and  Development,  and 
full-scale  production  programs,  the  DOD  requires  contractors 
to  submit  performance  measurement  information.  Performance 
measurement  information  is  produced  by  management  systems 
for  control  purposes  and  is  verified  as  valid  and  timely  by 
its  adherence  to  the  Cost /Schedule  Control  Systems  Criteria 
(C/SCSC) .  This  research  effort  examines  how  performance 
measurement  systems  are  evaluated  during  source  selection. 
This  thesis  explains:  the  C/SCSC-rel ated  items  of  a  DOD 
solicitation  and  the  resulting  requirements  placed  on  a 
contractor's  proposal;  how  the  source  selection  process 
works;  the  importance  of  C/SCSC  in  regard  to  the  overall 
source  selection;  and  how  the  contractor's  C/SCSC  plan  is 
evaluated  during  source  selection. 

The  research  is  based  primarily  on  interviews  with 
Aeronautical  Systems  Division  personnel.  Those  interviewed 
were  chosen  based  on  their  position,  experience,  and 
training.  The  general  agreement  among  those  interviewed 
provides  a  basis  for  developing  a  list  of  important  items  to 
consider  during  source  selection  when  evaluating  contracts 
which  include  C/SCSC  clauses. 


EVALUATION  OF  PROCEDURES  EMPLOYED  DURIN6  SOURCE 
SELECTION  FOR  CONTRACTS  INCLUDIN6  CLAUSES  REQUIRING 
COST/SCHEDULE  CONTROL  SYSTEMS  CRITERIA  (C/SCSC) 


I .  Introduct i on 


General  Issue 

The  Department  o-f  Defense  (DOD)  develops  and  procures 
many  items  that  are  not  available  in  the  consumer 
marketplace.  These  items  often  advance  technology,  use 
exotic  materials,  take  much  time  and  e-f-fort  to  develop,  am 
are  very  expensive  to  produce.  Additionally,  the  DOD 
marketplace  is  small  in  competitive  number  o-f  contractors. 
Only  a  few  large  contractors  can  produce  items  such  as 
aircraft,  ships,  tanks,  and  radars.  All  too  frequently 
these  conditions  have  led  to  huge  cost  overruns,  lengthy 

schedule  delays,  and  critical  reports  of  waste!  this 

•» 

has  caused  concern  among  the  public  and  the  government.  Ii 
order  to  help  the  government  monitor  and  control  the 
progress  of  military  acquisitions,  the  DGD  has  developed 
regulations  and  instructions  to  describe  the  various 
requirements  for  doing  business  with  the  military. 

Department  of  Defense  Directive  (DODD)  5000.1,  “Major 
System  Acquisitions,"  provides  the  guidance  for  large 
acquisitions  in  terms  of  time  and  dollar  effort.  This 
directive  is  supported  by  the  Air  Force  Systems  Command's 
(AFSC)  supplement  to  AFR  800-6.  This  regulation  requires 


that  contracts  in  excess  of  $40  million  -for  Research, 
Development,  Testing,  and  Engineering  (RDT&E)  effort  or  $160 
million  in  production  effort  must  contain  clauses  requiring 
the  contractor  to  adhere  to  Cost /Schedule  Control  Systems 
Criteria  (C/SCSC) .  The  government  can  also  require  C/SCSC 
on  smaller  efforts  with  a  value  of  at  least  $10  million 
<3:Atch  4).  Programs  that  require  C/SCSC  will  be  referred 
to  as  “major"  programs  throughout  this  thesis. 

When  the  government  awards  a  "major "  contract, 
performance  measurement  reporting  to  the  system  program 
office  (SPO)  is  required.  The  C/SCSC  concept  was  developed 
by  the  government  to  ensure  that  contractors  have  the 
ability  to  measure  and  report  performance  in  an  accurate, 
complete,  and  timely  fashion  (12:3).  The  C/SCSC  contain  35 
standards  (or  criteria),  all  of  which  must  be  satisfied  by 
the  contractor's  performance  measurement  system  (10x5). 

C/SCSC  is  established  by  Department  of  Defense 
Instruction  (DODI)  7000.2  titled  "Performance  Measurement 
for  Selected  Acquisitions'*  (lO).  The  stated  policy  of  DODI 
7000.2  is  to  provide  the  D0D  data  that  is  "derived  from  the 
same  internal  management  control  system  as  used  by  the 
contractor  to  manage  the  contract"  (12:3).  The  SPO  uses 
this  information  to  schedule  other  program  activities, 
identify  potential  problem  areas,  evaluate  the  cost  and 
schedule  impact  of  these  problems,  and  determine  future 
funding  requirements  (17:178). 
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Speci-fic  Problem 


The  contractor's  capacity  to  adhere  to  the  C'SCSC 
requirements  is  often  not  evaluated  adequately  durinq  source 
selection.  Even  though  regulations  have  required  C/SCSC 
since  1967,  many  procurement  officials  lack  familiarity  with 
C/SCSC  requirements.  Source  selection  committee  members 
have  difficulty  evaluating  contractor  proposals  because  the 
regulations  do  not  explain  exactly  what  to  look  for  while 
evaluating  the  proposals  (5). 

The  lack  of  specific  direction  makes  the  source 
selection  process  very  difficult  and  often  inadequate. 
Section  52.242-7001  of  the  DOD  supplement  to  the  Federal 
Acquisition  Regulation  <FAR)  directs  the  contractor  to 
submit  a  "Comprehensive  Plan  for  C/SCSC."  If  the  contractor 
already  has  a  validated  system  (validated  by  the  government 
against  the  35  criteria) ,  submitting  evidence  of  the 
validation  satisfies  the  requirement  (9:25647). 

By  submitting  a  Comprehensive  Plan  for  C/SCSC  in 
response  to  a  RFP  requiring  C/SCSC,  the  contractor  is 
certifying  that  their  performance  measurement  system 
satisfies  the  criteria.  This  plan  helps  the  government  in 
determining  if  the  contractor  understands  the  criteria.  The 
contractor's  performance  measurement  system  does  not  have  to 
actually  prove  adherence  with  the  C/SCSC  until  after 
contract  award  (5)  . 

It  is  important  to  evaluate  the  contractor's 


performance  measurement  system  in  source  selection.  The 
government  requires  C/SCSC  as  it  helps  both  the  government 
and  the  contractor  manage  the  program  better  (12:3).  The 
government  wants  to  receive  performance  measurement 
information  early  in  the  program  so  that  problems  can  be 
identified  and  corrected  before  a  situation  becomes 
critical.  By  evaluating  C/SCSC  during  source  selection  the 
government  reduces  the  probability  that  a  contractor  without 
a  proper  performance  measurement  system  will  be  awarded  a 
contract  (20).  If  the  contract  is  awarded  to  a  contractor 
who  has  an  inadequate  plan  for  C/SCSC,  performance 
measurement  may  be  inadequate  and  the  information  provided 
to  the  government  may  be  a  distortion  of  the  actual 
conditions  (17:94). 

Definitions 

The  following  terms  are  used  throughout  this  paper  and 
are  defined  below. 

Checklist.  A  checklist  is  a  listing  of  important  items 
that  are  stated  in  a  question  form  that  can  normally  be 
answered  in  a  "yes"  or  "no"  fashion.  Checklists  often  break 
down  complicated  procedures  or  questions  into  series  of 
straightforward  questions  that  can  be  easily  answered.  Most 
checklists  are  designed  so  that  either  “yes"  or  “no"  is  the 
preferred  answer  while  any  other  response  indicates  a 
potential  problem. 
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Work  Breakdown  Structure  (WBS) ■ 


“A  work  breakdown 


structure  is  a  product-orientated  family  tree  composed  of 

hardware,  software,  services,  and  data"  (11*2).  The  WBS 

completely  defines  the  project  or  contract  by  dividing  the 

product  into  detailed  levels  of  components  that  add  up  to 

the  total  product.  The  WBS  identifies  all  the  elements  that 

make  up  the  total  product  and  shows  how  the  elements 

interrelate.  Common  types  of  WBS  are  Summary  WBS,  the  upper 

three  levels  only;  Contract  WBS,  contains  the  activities  of 

a  specified  contract;  and  Project  WBS,  contains  the 

contracted  events  along  with  all  other  activities  that  must 

be  completed  within  the  project.  Military  Standard  881A, 

Work  Breakdown  Structures  for  Def ense  Material  Items. 

provides  guidance  for  work  breakdown  structures  and  gives 

examples  of  Project  Summary  WBSs  (11). 

Source  Selection.  Source  selection  is  a  highly 

structured  process  used  to  determine  which  contractors 

should  be  awarded  contracts.  Guidance  for  source  selection 

is  provided  by  AF  Regulation  (AFR)  70-15,  "Source  Selection 

Policy  and  Procedures. "  According  to  this  regulation  the 

objective  of  source  selection  is* 

To  select  the  source  whose  proposal  has  the 
highest  degree  of  credibility  and  whose 
performance  can  be  expected  to  best  meet  the 
government's  requirements  at  an  affordable  cost. 

The  process  must  provide  an  impartial,  equitable, 
and  comprehensive  evaluation  of  the  competitors* 
proposals  and  related  capabilities.  The  process 
should  be  accomplished  with  minimum  complexity  and 
maximum  efficiency  and  effectiveness.  It  should 
be  structured  to  properly  balance  technical. 


financial,  and  economic  or  business  considerations 
consistent  with  the  phase  of  the  acquisition, 
program  requirements,  and  business  and  legal 
constraints.  It  must  be  sufficiently  flexible  to 
accommodate  the  objectives  of  the  acquisition  and 
a  decision  must  be  compatible  with  program 
requirements ,  risks,  and  conditions  C8:33. 

Specifics  on  the  role  that  C/SCSC  related  areas  play  in  the 

overall  source  selection  process  will  be  described  in  later 

chapters. 


Research  Objective 

The  research  objective  is  to  investigate  how  the  source 
selection  community  determines  that  a  contractor's  proposal 
satisfies  the  C/SCSC-rel ated  areas  of  the  solicitation. 
Research  will  also  look  at  the  options  available  to  the 
government  when  a  contractor's  response  is  unacceptable. 

The  product  of  this  research  will  be  a  list  of  items  to 
consider  when  evaluating  C/SCSC  during  source  selection. 


Research  Questions 

1.  What  type  of  information  concerning  C/SCSC  is  the 
contractor  required  to  provide  to  the  government  in  a 
proposal ? 

2.  How  are  contractors'  proposals  evaluated  during 
source  selection? 

3.  How  important  are  the  C/SCSC-rel ated  areas  in  the 
overall  source  selection? 

4.  What  specific  elements  should  the  persons 
evaluating  the  Comprehensive  Plans  for  C/SCSC  examine  in  the 
proposal s? 


II. 


Literature  Review 


Topic  Statement 

This  chapter  examines  published  literature,  government 
publications,  and  seminar  presentations  that  deal  with  the 
Cost /Schedule  Control  Systems  Criteria. 

Just if i cat ion 

It  is  important  to  understand  the  C/SC SC  in  order  to 
understand  the  problems  of  evaluating  C/SCSC  during  source 
selection.  Looking  at  available  literature  provides  a  basis 
for  determining  the  depth  of  information  that  is  available 
and  the  areas  that  need  to  be  researched  further  in  order  to 
satisfy  the  research  objective. 

Scope 

Much  of  the  information  on  C/SCSC  is  published  by  the 
DOD  in  the  form  of  regulations,  directions,  and  guides. 
Besides  examining  government  publications  and  published 
literature,  this  review  also  looks  at  detailed  studies  and 
presentations  from  a  C/SCSC  conference.  Research  is  limited 
to  materials  available  through  the  Air  Force  Institute  of 
Technology  (AFIT)  libraries.  (AFIT  is  the  lead  agency, 
appointed  by  the  OSD  for  educating  DOD  agencies  and 
organizations  on  C/SCSC  (2>). 
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Pi scussion 

Three  areas  concerning  the  C/SCSC  will  be  discussed. 

The  -first  area,  "Purposes  and  Objectives  o-f  C/SCSC," 

establishes  the  reasons  why  C/SCSC  was  established.  The 

second  area,  "Criteria  and  the  Implementation  Process," 

presents  the  content  o-f  the  criteria.  The  -final  area, 

“Problems  and  Perceptions  of  C/SCSC,"  identifies  some  of  the 

difficulties  inherent  in  the  C/SCSC  concept. 

Purposes  and  Objectives  of  C/SCSC.  Due  to  the 

importance  and  high  cost  of  major  projects,  it  is  very 

important  that  a  contractor  have  a  performance  measurement 

system  (for  cost  and  schedule  control )  that  provides 

accurate,  timely  information  for  decision-making  (10:3). 

However,  Marvin  Friedenberg,  head  of  the  Cost  Performance 

Management  Branch  at  the  Naval  Material  Command  states: 

Cost  control  [performance  measurement!  systems  do 
not  control  cost.  People  control  cost.  Cost 
control  systems  aid  people  in  controlling  cost  by 
providing  pertinent,  useful  information  in  a 
readily  understandable  form  in  a  timely  manner. 

No  control  system,  no  measurement  of  performance, 
no  report  of  status  is  ever  an  end  in  itself 
C 13: 2571 . 

DODD  7001 ,  "Resource  Management  Systems  of  the 
Department  of  Defense,"  among  other  things,  identifies  the 
need  for  contractors  to  have  an  "effective"  performance 
measurement  system  for  providing  accurate  and  timely 
information.  To  further  identify  the  intrinsic  attributes 
of  an  "effective”  management  system,  the  DOD  developed  DODI 
7000.2,  "Performance  Measurement  for  Selected  Acquisitions.” 


This  instruction  contains  two  main  abjectivesi 


1.  To  provide  an  adequate  basis  -for 
responsible  decision-making  by  both  contractor 
management  and  DOD  components,  contractors' 
internal  management  control  systems  must  provide 
data  which  (a)  indicate  work  progress,  (b) 
properly  relate  cost,  schedule,  and  technical 
accomplishment,  (c)  are  valid,  timely,  and 
auditable,  and  (d)  supply  DOD  managers  with 
information  at  a  practicable  level  of 

summar i  z  a  t  i  on . 

2.  To  bring  to  the  attention  of,  and 
encourage  DOD  contractors  to  accept  and  install 
management  control  systems  and  procedures  which 
are  cost  effective  in  meeting  their  requirements 
and  controlling  contract  performance.  DOD 
contractors  also  should  be  continuously  alert  to 
advances  in  management  control  systems  which  will 
improve  their  internal  operations  ClOt 11. 

These  objectives  imply  that  the  criteria  were  designed 

to  help  both  industry  and  government.  Sary  Christie,  from 

the  Office  of  the  Assistant  Secretary  of  Defense 

(Comptrol ler ) ,  has  identified  seven  acquisition  management 

policies  of  the  DOD  that  were  used  in  developing  the 

cri teria. 

(1)  Reduce  &  standardize  requirements 
<2)  Make  maximum  use  of  contractor's 
[performance  measurement]  system 

(3)  Review  contractor's  system  for  adequacy 

(4)  Avoid  imposing  specific  methods  and 
techni ques 

<5>  Tailor  reporting  requirements  to  fit 
programs 

<6)  Use  common  work  breakdown  structure 
<7>  Obtain  summary  level  data  C6:783. 

Robert  R.  Kemps,  from  the  Department  of  Energy  and 

former  Air  Force  C/SCSC  focal  point,  adds  that  a  cost  and 

schedule  control  system  should  "insure  a  single  system  for 

internal  management  and  reporting  to  the  government"  and 
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"insure  that  reported  data  relates  contract  performance 
accurately  and  objectively”  (17:96). 

The  C/SCSC  process  is  intended  to  better  DOD's 
management  of  large  programs.  L.  A.  Stone,  from  the  Army's 
Contract  Cost  Management  Division,  gave  a  presentation  at  a 
conference  sponsored  by  the  American  Institute  of 
Aeronautics  and  Astronautics  in  which  he  listed  seven 
benefits  to  the  government  from  contractors  that  have  a 
validated  performance  measurement  system: 

(1)  [government  gains!  Confidence  in 
contractor's  internal  management  system. 

(2)  Measurement  against  a  controlled,  contract- 
orientated  baseline. 

<3)  Objective  (rather  than  subjective)  contract 
status  information. 

(4)  Cost  impact  of  known  problems. 

(5)  Identification  of  problems  not  previously 
recognized. 

(6)  Capability  to  trace  problems  to  source 
(hardware  and  organizational). 

(7)  Quantitative  measure  of  schedule  deviation 
in  dollars  [21:234!]. 

Besides  benefiting  the  government,  a  validated 
performance  measurement  system  can  be  a  big  advantage  to 
industry.  In  his  presentation.  Stone  listed  six  benefits  to 
contractors  from  having  a  performance  measurement  system 
that  meets  the  C/SCSC  requirements. 

<1)  System  discipline 

(2)  Earned  value 

(3)  Improved  planning 

<4)  Better  communication 

(5)  Deeper  and  earlier  program  visibility 

(6)  Cost  and  schedule  awareness  [21:2341. 


> 
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Criteria  and  the  Implementation  Process 


Cost/Schedule  Control  Systems  Criteria  described  by  DODI 
7000.2  consists  o-f  35  separate  criteria  broken  down  into 
five  sections.  The  five  sections  are  "organization," 
"planning  and  budgeting,"  "accounting,"  “analysis,"  and 
"revisions  and  access  to  data"  <10i9-12).  The  criteria  are 
general  in  nature  and  evaluation  against  the  criteria  often 
requires  subjective  judgments. 

The  "Evaluation/Demonstration  Review  Checklist  for 
C/SCSC"  (Attachment  A)  is  used  to  evaluate  the  contractor's 
adherence  to  the  criteria  (12:9-12).  This  checklist  breaks 
down  each  criteria  into  a  series  of  questions  that  aids  in 
the  evaluation  of  the  contractor's  performance  measurement 
system. 

The  Cost /Schedul e  Control  Systems  Criteria  Joint 
Implementation  Guide,  published  under  the  auspices  of  the 
Office  of  the  Assistant  Secretary  of  Defense  (Comptrol ler ) , 
provides  interpretive  information  on  the  criteria.  This 
guide,  used  by  all  three  military  services  and  the  Defense 
Logistics  Agency,  contains  a  detailed  explanation  of  each 
criterion  along  with  a  discussion  of  appropriate 
implementation  procedures  (12). 

According  to  these  procedures,  a  contractor  does  not 
need  an  "approved"  performance  measurement  system  to  be 
awarded  a  government  contract.  FAR  clause  52.242-7002 
allows  the  contractor  90  days  after  contract  award  (or 


longer  if  agreed  to  by  the  government)  before  a 


Demonstration  of  C/SCSC  compliance  is  required.  A 
Demonstration  Review  of  the  contractor's  performance 
measurement  system  is  conducted  by  the  procuring  activity  to 
ensure  that  all  35  criteria  are  satisfied.  The  performance 
measurement  system  is  said  to  be  "validated"  when  all  the 
criteria  have  been  satisfied  (9:25647-8). 

In  evaluating  a  performance  measurement  system  against 
the  criteria,  the  C/SCSC  Joint  Implementation  Guide  cautions 
that  "use  of  the  criteria  must  be  based  on  common  sense, 
which  means  being  practical,  but  also  mindful  of  the  overall 
requirements  for  performance  measurement"  (12:3). 

If  the  contractor  already  has  an  approved  system,  a 
Subsequent  Application  Review  (SAR)  may  be  performed  in  lieu 
of  a  Demonstration  Review.  This  review  ensures  "the 
contractor  is  properly  and  effectively  using  the  accepted 
system,  revised  in  accordance  with  approved  changes” 

(12:66).  These  reviews  ensure  that  both  government  and 
industry  receive  the  benefits  of  good  performance 
measurement . 

A  primary  benefit  of  C/SCSC  is  knowing  the  current 
status  of  all  elements  of  the  program.  C/SCSC  requires  the 
current  status  to  be  reported  to  the  government  via  a  Cost 
Performance  Report  (CPR)  on  a  routine  and  timely  basis. 

This  report  examines  every  WBS  item  and  lists  for  each:  the 
amount  of  work  scheduled,  the  amount  of  work  completed,  the 
budgetary  goal  for  the  work,  and  its  actual  cost  (12:19). 
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This  report  also  identities  potential  problem  areas  by 
reporting  variances.  A  variance  is  the  difference  between 
what  was  planned  and  what  actually  occurred.  When  a 
variance  is  significant,  the  contractor  reports  the  cause  of 
the  problem  to  the  government  and  identifies  corrective 
actions  that  have  been  initiated.  This  information  is 
useful  in  effectively  managing  the  program  (12:18-9). 

Problems  and  Perceptions  of  C/SCSC.  Despite  the 
benefits  mentioned  earlier,  many  contractors  view  C/SCSC  as 
"a  four — letter  word,  ranking  just  slightly  above 
'technically  unacceptable'  and  'defective  pricing*  in 
popularity*1  (24:11).  This  section  on  problems  and 
perceptions  of  C/SCSC  identifies  some  of  the  reasons  why 
there  is  contractor  resistance  to  the  C/SCSC  concept.  This 
section  may  also  help  explain  why  there  are  problems  in 
evaluating  contractors'  proposals  during  source  selection. 

Contractors  have  many  reasons  for  disliking  C/SCSC 
requirements.  A  1983  article,  "Taking  the  Heartburn  out  of 
CS2"  written  by  Varady  &  Lumer ,  mentions  some  of  the 
reasons: 

(1)  Lack  of  incentive  (money) 

(2)  Cost  involved  with  setting  up  system 

(3)  Lack  of  knowledge  and  understanding 

(4)  Ability  to  get  contracts  without  system 

(5)  Corporate  dislike  for  government  procedures 

(6)  Lack  of  ability  C24:113. 

Another  1983  article,  "The  Cost  of  C/SCSC"  by  Gadeken 
and  Tison,  discusses  the  cost  and  benefits  of  C/SCSC.  Many 
contractors  and  some  DOD  managers  question  the  cost  of 


implementing  a  C/SCSC  system  compared  to  the  benefits 

received.  In  their  opinion,  a  large  problem  is  how  to 

quantify  the  cost  and  benefits  of  C/SCSC. 

The  benefits  of  C/SCSC  are  good  cost /schedul e 
control  of  a  project,  but  quantifying  such 
benefits  is  the  same  as  trying  to  answer  the 
question  'What  is  the  value  of  good  management?' 

Even  in  an  isolated  analysis  of  a  specific 
contract,  benefits  of  using  C/SCSC  cannot  be 
clearly  associated  with  results  in  terms  of 
improved  technical  performance,  increased  quality, 
shortened  schedule,  or  contract  cost  savings,  let 
alone  quantified  into  one  cost  figure  114:15]. 

Gadeken  &  Tison  explain  that  determining  the  cost 

associated  with  C/SCSC  is  almost  impossible.  Many  of  the 

reports  and  procedures  required  by  C/SCSC  would  be  part  of 

any  normal  management  system.  The  cost  of  implementing 

C/SCSC  for  the  first  time,  therefore,  often  includes  bias 

based  on  contractors'  views  of  C/SCSC.  While  there  is  a 

cost  involved  with  implementation,  the  quantification  of 

this  cost  is  infeasible  because  of  factors  such  as  "initial 

company  management  systems,  company  size,  contract  value, 

government  criteria  interpretation,  and  atti tude/ acceptance" 

( 14: 16) . 

Gadeken  &  Tison  also  point  out  problems  with  the 
implementation  process.  The  government  team  that  reviews 
the  contractor’s  performance  measurement  system  often 
includes  individuals  without  proper  knowledge  and  experience 
on  C/SCSC.  These  people  have  problems  making  subjective 
judgments  when  the  contractor's  system  appears  not  to 
exactly  satisfy  the  criteria.  In  some  cases,  these 


judgments  have  resulted  in  the  contractor  being  cited  -for 
noncompliance.  "Rigid  interpretation  o-f  the  criteria  is  the 
cause  'of  continuing  disagreement  between  the  DOD  and 
industry"  (14:17). 

Even  though  all  contractors  are  not  totally  in  favor  of 
C/SCSC,  "the  worth  of  C/SCSC  to  DOD  managers  has  been  proved 
over  the  years”  (14:18).  Recent  highly  publicized  reports 
of  questionable  billing  practices  at  major  defense 
contractors  including  General  Electric  and  General  Dynamics 
have  caused  Congress  and  the  American  public  to  question  the 
DOD's  relationship  with  the  defense  industry.  The  8  April 
1985  edition  of  Time  has  a  section  on  current  problems  with 
DOD  contractors.  In  one  article,  a  Pentagon  spokesman 
explains  that  the  DOD  must  do  a  better  job  monitoring  and 


controlling  cost  on  major  contracts  in  order  to  "restore  the 
public  support  for  a  stronger  defense  that  has  been 
squandered  by  repeated  revelations  of  waste,  fraud,  and 


abuse"  (23:20). 

The  Arthur  D.  Little  Program  Systems  Management  Company 
performed  a  detailed  study  called  Survey  Relating  to  the 
Implementation  of  Cost/Schedule  Control  Svstems  Criteria 


within  the  Department  of  Defense  and  Industry.  This  study 
which  was  commissioned  by  the  Office  of  the  Secretary  of 


Defense  (OSD)  and  completed  in  1983,  surveyed  program  and 
business  managers  from  the  DOD  and  the  defense  industry. 
It  is  viewed  by  many  experts  as  the  most  complete  C/SCSC 
study  done  to  date  (5) . 


Several  summarized  -findings  from  this  survey  include: 


(1)  Government  review  teams  lack  flexibility 
and  technical  qualifications. 

<2)  Review  teams  do  not  consistently  interpret 
C/SCSC  during  demonstration  reviews. 

(3)  Demonstration  reviews  often  take  longer  than 
six  months. 

(4)  CPRs  are  received  to  late  to  provide  timely 
data. 

(5)  Program  offices  do  not  perform  timely  reviews 

of  CPRs. 

<6)  Government  contract  administration 

representatives  lack  expertise  in  C/SCSC 
C  4: IV-5-13D. 

The  Little  study  also  identifies  some  positive  points 
about  the  use  of  C/SCSC.  Those  surveyed  feel:  (1)  that 
despite  the  problems,  the  benefits  from  C/SCSC  outweigh  the 
cost;  and  (2)  that  the  criteria  approach  is  as  good  or 
better  than  other  methods  of  ensuring  adequate  performance 
measurement  (4: IV— 2— 4) . 

A  big  problem  for  contractors  is  the  implementation 
process.  Gadeken  &  Tison  feel  this  problem  could  be  solved 
by  having  government  personnel  experienced  with  C/SCSC  work 
closely  with  the  contractor.  The  government  personnel 
should  start  communications  early,  explain  the  required 
terminology  and  concepts,  and  understand  the  peculiarities 
of  the  contractor's  system.  This  effort  will  use  a  great 
deal  of  the  government's  limited  C/SCSC  resources.  However, 
the  benefits  of  accurate  performance  measurement  information 
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early  in  the  program  should  outweigh  the  cost  involved 
(14: 17) . 

Varady  and  Lumer  take  a  different  view  on  how  to  get 
contractors  to  accept  C/SCSC.  They  believe  the  government 
should  forcefully  motivate  the  contractor  to  want  to  use  a 
C/SCSC  validated  system.  They  list  three  approaches  for  a 
timely  implementation: 

1.  An  award  fee  for  implementation  and 
management  of  the  CS  CC/SCSC1  program 
throughout  the  life  of  the  contract. 

2.  A  formal  course  of  instruction  for  key 
contractor  personnel ,  to  be  set  forth  as 
a  contractual  requirement. 

3.  A  withholding  of  (any)  fee  until  CS 
CC/SCSC1  is  fully  implemented  C24illl. 

If  the  contractor  still  does  not  implement  C/SCSC, 

Varady  &  Lumer  suggest  a  clause  in  the  contract  that  allows 

for  contract  termination  if  the  contractor  refuses  to 

establish  an  acceptable  performance  measurement  system 

(24: 12) . 

Results  of  Literature  Review 

Much  has  been  written  on  C/SCSC  both  pro  and  con.  In 
addition,  courses  and  seminars  taught  by  both  the  government 
and  private  sector  are  aimed  at  giving  people  a  better 
understanding  of  the  C/SCSC  concepts.  This  information 
covers  purposes  and  objectives  of  C/SCSC,  the  criteria 
themselves  and  their  implementation  process,  and  problems 
and  perceptions  of  C/SCSC.  This  literature  review 
has  uncovered  valuable  information  on  many  aspects  of 
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I 

> 

C/SCSC.  However,  it  has  reflected  a  lack  of  substantial 
information  and  guidance  for  determining  acceptability  of 

i 

contractor  s  plans  for  a  performance  measurement  system  in 
source  selection.  In  other  words,  there  is  a  need  to 
investigate  how  the  source  selection  community  determines  if 
a  contractor's  proposal  satisfies  the  C/SCSC  related  areas 
of  the  solicitation. 
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III. 


Methodo I ogy 

This  chapter  describes  the  methodology  used  to  solve 
the  research  problem.  The  research  on  this  project  consists 
o-f  two  phases.  The  initial  phase  concerns  researching  the 
requirements  o-f  the  Cost /Schedul e  Control  Systems  Criteria 
(C/SCSC)  and  the  source  selection  process.  Secondly,  the 
primary  phase  of  the  research  explores  how  the  government 
evaluates  C/SCSC  during  the  source  selection  process. 

Initial  Research  Phase 

The  first  step  in  this  research  is  to  gain  an 
understanding  of  C/SCSC.  Initial  Orientation  to  C/SCSC  was 
accomplished  by  attending  the  professional  continuing 
education  course  SYS  362  “Cost /Schedule  Control  Systems 
Criteria"  at  the  Air  Force  Institute  of  Technology  in 
January  19B4. 

The  literature  review  (chapter  2)  examined  official 
direction  and  guidance  along  with  opinions  and  findings  from 
knowledgeable  people  in  the  C/SCSC  field.  This  review 
resulted  in  several  findings  that  otherwise  would  have  to  be 
included  in  the  research.  The  literature  review  also 
verified  that  research  was  lacking  in  regards  to  determining 
how  the  C/SCSC  related  areas  are  examined  during  source 
selection. 

Interview  questions  were  formulated  using  the  AFIT 
course  and  literature  review  as  background.  Personnel  at  Air 


Force  Systems  Command  <AFSC) ,  the  Air  Force's  R&D  and 
production  procuring  organization,  were  interviewed  -first  to 
determine  command  guidelines  and  policies.  Division  heads 
-from  Aeronautical  Systems  Division  (ASD)  were  also  asked 
similar  questions.  (ASD  is  one  o-f  the  Air  Force's  product 
divisions.  A  product  division  is  a  separate  procuring 
organization  that  specializes  in  a  certain  area.  Other 
product  divisions  include  Electronic  Systems  Division  (ESD) , 
Space  Division  (SD) ,  and  Armament  Division  (AD) ) .  The 
questions  asked  in  these  preliminary  interviews  changed  as 
more  was  learned  about  C/SCSC. 

Lt  Col  Connie  Teetz ,  chief  of  AFSC's  Cost  Information 
Division  (AFSC/ACCI),  explained  in  a  telephone  interview 
that: 

(1)  There  is  not  an  AFSC  focal  point  in  charge  of 
source  selections. 

(2)  The  different  product  divisions  can  require 
contractors  to  submit  different  information 
but  the  C/SCSC  requirements  must  be 
consistent  with  the  FAR  and  AFR  70-15. 

(3)  There  is  not  an  AFSC  guideline  for 
evaluating  C/SCSC  during  source  selection. 

However,  based  on  Lt  Col  Teetz 's  frequent 
conversations  with  the  product  divisions, 
there  is  little  difference  in  the  way  the 
product  division's  evaluate  C/SCSC  during 
source  selection  (22). 

These  points,  which  were  verified  by  second  sources  at  both 
AFSC  and  ASD,  helped  to  determine  where  to  focus  the 
remaining  research  effort. 
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Initially,  source  selection  and  C/SCSC  personnel  from 
the  different  product  divisions  were  to  be  interviewed, 
however,  this  proved  to  be  unnecessary.  Based  on  the 
preliminary  interviews  it  was  discovered  that  the  product 
divisions  can  organize  and  evaluate  source  selections 
differently  but  they  still  must  follow  the  procedures  of  AFR 
70-15.  Therefore,  any  product  division  could  be  considered 
as  "typical"  and  ASD  was  chosen  because  of  it's  close 
proximity  to  AFIT. 

The  source  selection  process  was  explained  in  detail  by 
James  Helmig,  chief  of  the  Source  Selection  Division  at  ASD 
(ASD/PMPS) .  The  findings  of  this  interview  are 
substantiated  by  the  ASD  handbook.  The  Source  Selection 
Process.  An  attempt  to  witness  a  source  selection  was 
disallowed  due  to  the  sensitive  nature  of  the  source 
selection  process. 

Primary  Research  Phase 

From  the  initial  interviews,  ASD/ACCM  was  identified  as 
the  office  that  evaluates  the  C/SCSC  area  during  source 
selections  at  ASD.  Currently  at  ACCM,  Patrick  Cyrus, 

William  Harper,  John  Wohlman,  and  Stanley  Nesivich  (the 
division  chief)  are  the  only  people  who  have  evaluated 
contractors'  performance  measurement  systems  during  source 
selection.  Another  person  in  the  ASD  community  with 
experience  in  this  area  is  Dan  Schild  from  The  Analytical 
Sciences  Corporation  (TASC).  Schild  was  the  division  chief 


of  ASD/ACCM  before  working  as  a  C/SCSC  consultant  with  TASC. 
These  people  were  interviewed  because  their  organi zational 
responsibilities  and  job  prerequisites  require  them  to  be 
experienced  and  trained  in  C/SCSC  and  source  selection. 

These  five  people  were  interviewed  in  their  own  offices 
at  separate  times  to  elicit  their  individual  opinions.  To 
insure  that  the  information  provided  would  be  valid  and 
useful ,  the  purpose  of  the  interview  was  explained  and 
anonymity  was  offered.  All  persons  answered  the  questions 
freely.  Records  of  the  interviews  are  attached  in 
Appendix  B. 

Chapter  4  presents  the  findings  of  the  interviews.  In 
most  cases,  those  interviewed  expressed  the  same  views.  In 
these  instances,  special  points  brought  out  during  the 
interview  are  referenced  to  the  appropriate  individual. 

Since  nonattribution  was  offered,  individuals  were  not 
referenced  in  the  text  in  those  cases  where  consensus  was 
not  reached  between  the  interviewees.  In  these  cases, 
paraphrased  and  quoted  information  from  the  interviewees  are 
represented  by  " <*> "  in  the  text.  These  areas  are  deleted 
from  the  record  of  interviews. 

A  list  of  C/SCSC-rel ated  items  to  consider  in  source 
selection  was  developed  prior  to  interviewing  the  five 
C/SCSC  evaluators.  At  the  end  of  each  interview,  the 
interviewee  was  shown  the  list  and  comments  were  solicited. 
The  comments  were  incorporated  before  trie  next  individual 
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was  interviewed.  Due  to  general  approval  of  the  list  by  the 
final  few  interviewees,  the  evaluators  were  not  asked  to 
examine  the  list  a  second  time.  This  list  is  presented  in 
the  conclusion  (chapter  5). 

Reasons  for  Methodology  Used 

People  with  experience  in  both  C/SCSC  and  source 
selection  were  interviewed  in  order  to  determine  how  C/SCSC 
is  evaluated  during  source  selection.  It  was  possible  to 
interview  the  entire  population  of  an  acquisition 
organization  due  to  the  small  number  of  people  who  have 
experience  evaluating  the  C/SCSC  area  in  proposals. 

The  interview  process  allowed  the  use  of  open-ended 
questions  which  are  useful  for  having  someone  explain  a 
process.  Interviews  also  allow  for  questions  which  clarify 
and  verify  information  received  from  others.  Analyzing  the 
interviews  for  agreement  and  differences  helps  explain  the 
process  of  evaluating  the  C/SCSC-rel ated  areas  of  a 
contractor's  proposals  during  source  selection. 

Sending  a  survey  to  other  procuring  agencies  was 
considered.  There  are  often  problems  inherent  in  surveys. 
These  problems  include  wording  the  questions  for  consistent 
interpretation,  correctly  understanding  the  responses,  and 
controlling  the  accuracy  of  the  responses.  Since  the 
initial  research  indicated  that  procuring  organizations 
evaluate  C/SCSC  similarly  during  source  selection  (22), 
interviewing  one  organization  was  chosen  over  surveying  all 


DOD  personnel  with  experience  in  evaluating  C/SCSC  during 
source  selection. 

It  was  unnecessary  (and  impractical)  to  quantify  the 
interviewees'  responses  as  the  interviews  were  open-ended. 
Therefore,  this  thesis  contains  no  statistical  analysis. 
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IV 


Findings 


This  chapter  contains  the  information  obtained  from 
interviewing  selected  personnel  at  AFSC  &  ASD.  The  findings 
of  this  research  are  organized  as  answers  to  the  research 
questions  posed  at  the  end  of  chapter  one.  The  interview 
responses  for  question  four  are  also  compared  and 
contrasted. 

Research  Question  1. 

1.  What  type  of  information  concerning  C/SC8C  is  the 
contractor  required  to  provide  to  the  government  in  a 
proposal ? 

At  ASD,  all  Requests  for  Proposal  <RFP>  must  be 
reviewed  by  the  Cost  Management  Systems  Division  (ASD/ACCM) 
before  the  RFP  is  released  to  contractors.  John  Wohlman,  a 
member  of  this  division  who  has  reviewed  many  RFPs, 
explained  that  the  System  Program  Office  (SPO)  designs  the 
RFP  to  meet  the  needs  of  the  program.  However,  the  FAR 
requires  certain  clauses  to  be  included  in  the  proposal  when 
dollar  thresholds  are  exceeded,  regardless  of  SPO  needs.  As 
mentioned  earlier,  C/SCSC  is  required  on  all  ••major" 
programs.  ASD/ACCM  reviews  the  RFP  for  all  such  programs  to 
ensure  that  DOD  FAR  Supplement  clause  52.242-7001,  "Notice 
of  Cost/Schedule  Control  Systems,"  is  included  as  part  of 
the  RFP  package  <25) . 

DOD  FAR  Supplement  Clause  52.242-7001  requires 
contractors  submitting  an  offer  to  include  a 


"Comprehensive  Plan  -for  C/SCSC"  with  their  proposal .  This 
plan  explains  the  contractor's  performance  measurement 
system  and  how  the  system  will  be  applied  to  manage  the  work 
authorized  in  the  contract.  The  plan  must  also  describe  how 
the  contractor's  performance  measurement  system  satisfies 
each  of  the  35  criteria  established  in  DODI  7000.2.  The  DOD 
uses  this  clause  to  notify  contractors  that  C/SCSC,  as 
discussed  in  DODI  7000.2,  is  required  on  the  contract.  If 
the  contractor  is  currently  using  an  accepted  system, 
evidence  of  the  DOD's  C/SCSC  validation  can  be  submitted 
instead  of  the  Comprehensive  Plan  for  C/SCSC  (9:25647). 

FAR  clause  52.242-7001  also  requires  the 
contractor  to  identify  critical  subcontractors  and  the 
effort  that  these  subcontractor  will  perform.  In  addition, 
this  clause  requires  the  contractor  to  "provide  information 
and  assistance  as  requested  by  the  Contracting  Officer  for 
evaluation  of  compliance  with  the  cited  crit  r ia"  (9:25647). 

In  addition  to  the  requirements  of  DOD  FAR  Supplement 
clause  52.242-7001,  the  contractor  also  must  submit  other 
C/SCSC-related  items.  Mr.  Wholman  explained  that  in 
procurements  requiring  C/SCSC,  the  RFP  includes  a  Project 
Summary  WBS  (broken  down  to  level  3)  and  instructions  to  the 
contractor  to  extend  this  WBS  to  the  cost  account  level  in 
their  proposal.  The  Contract  Work  Breakdown  Structure 
(CW9S)  submitted  by  the  contractor  must  reflect  how  the 
contractor  plans  to  complete  the  program.  The  contractor 
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must  also  develop  a  "CWBS  Dictionary”  which  de-fines  the 
scope  of  work  included  in  each  CWBS  element  <25) .  Lt  Col 
Thomas  Bowman,  who  instructs  C/SCSC  courses  at  AFIT,  adds 
that  "a  CWBS  Index  (which  is  a  matrix  of  which  CWBS  elements 
comprise  the  Contract  Line  Items  and  End  Items)  is  also 
required.  The  CWBS,  index,  and  dictionary  are  all  required 
via  a  single  'Data  Item  Description'”  (5). 

DOD  FAR  Supplement  Clause  52.242-7002,  ”Cost/Schedul e 
Control  Systems,"  is  the  clause  used  in  the  contract  itself 
to  follow-up  on  the  requisites  established  by  52.242-7001  in 
the  solicitation.  It  requires  the  contractor  to  demonstrate 
the  performance  measurement  system  that  was  described  in 
response  to  FAR  clause  52.242-7001  (9:25647-8).  Even  though 

FAR  clause  52.242-7002  is  not  evaluated  in  source  selection, 
it  is  still  important.  Daniel  Schild,  a  C/SCSC  consultant 
with  The  Analytic  Sciences  Corporation  (TASC)  and  former 
chief  of  ACCM,  explained  that  this  clause  forces  the 
contractor  to  plan  for  a  Demonstration/Validation  Review  or 
Subsequent  Application  Review  shortly  after  contract  award. 
Since  this  clause  requires  the  contractor  to  use  the  system 
described  in  the  Comprehensive  Plan  for  C/SCSC,  it  provides 
some  assurance  that  the  contractor  has  described  the  system 
it  intends  to  use  and  that  it  has  not  copied  the  performance 
measurement  system  plan  of  another  contractor  (20). 
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Research  Question  2. 


How  are  contractors’  proposals  svaluatsd  during  source 
selection? 

In  order  to  explain  the  role  C/SCSC  plays  in  source 
selection  it  is  essential  to  understand  what  takes  place 
during  this  process.  Mr  James  Helmig,  Chief  of  the  Source 
Selection  Division  at  ASD  (ASD/PMPS) ,  explained  the  Source 
Selection  elements  and  process  during  a  personal  interview. 

The  Source  Selection  Elements.  The  contractor  submits 
the  proposal  as  a  set  of  volumes.  Each  volume  contains 
information  that  is  needed  for  a  formal  proposal  evaluation 
and  each  is  subdivided  into  areas.  Normally,  there  are  five 
areas:  "Operational  Utility,"  "Readiness  Support,"  "Life 
Cycle  Cost,"  "Design  Approach,"  and  “Manufacturing/  Program 
Management"  (16). 

Mr.  Helmig  explained  that  each  of  these  areas,  except 
Life  Cycle  Cost,  are  subdivided  into  items.  There  are 
approximately  8-10  items  in  each  area.  Each  item  is  further 
subdivided  into  factors.  In  some  rare  instances,  the 
factors  are  broken  down  into  subfactors.  The  proposal  is 
thus  subdivided  into  smaller  elements  to  allow  for  simpler 
and  more  precise  evaluation  of  the  proposals  (16).  (The 
evaluation  process  is  described  in  the  next  section.) 

The  areas,  items,  and  factors  for  all  but  the  Life 
Cycle  Cost  area  are  determined  and  stated  in  the  "Source 
Selection  Plan  (SSP) . "  The  SPO  tailors  the  SSP  to  meet  the 
needs  of  the  program.  Therefore,  the  SSP,  especially  the 


items  and  -factors,  differ  from  program  to  program  (16). 

Other  attributes  of  the  SSP  are  the  evaluation  standards, 
source  selection  organization,  source  selection  schedule, 
and  program  peculiarities  that  are  relevant  to  source 
selection  (1:21-28). 

As  mentioned  earlier,  the  Life  Cycle  Cost  (LCC>  area  is 
not  broken  down  into  items  and  factors.  Donald  Osborne, 
chief  of  the  Resources  Validation  Division  (ACCC) ,  explains 
that  at  ASD  his  ACCC  division  and  SPO  Program  Control 
personnel  jointly  evaluate  the  LCC  area.  This  valuation  is 
to  determine  whether  the  bid  submitted  by  the  contractor  is 
realistic  and  contains  all  costs.  ACCC  and  the  SPO 
determine  the  operating  and  support  cost  for  the  entire  life 
of  the  program  based  on  the  contractor ' s  design  approach. 

The  LCC  area  is  then  summarized  into  a  "most  probable  cost" 
for  the  program  (19). 

The  source  selection  process  (including  the  source 
selection  plan  referred  to  earlier)  is  explained  in  detail 
in  the  ASD/PMPS  handbook.  The  Source  Selection  Process. 
According  to  this  handbook,  the  SPO  develops  evaluation 
standards  concurrently  with  the  development  of  the  Source 
Selection  Plan.  These  standards  are  used  to  determine 
whether  the  contractor’s  response  satisfies  the  individual 
factors.  A  standard  normally  represents  the  minimal  quality 
acceptable  (1:28-31). 

The  handbook  also  discusses  the  hierarchy  of  a  source 


selection  organization.  The  evaluation  of  the  factors  is 
typically  done  by  non -super v i sory  people  from  the  SPO  and 
various  staff  offices  (e.g.,  comptroller,  manufacturing, 
test,  etc.).  These  people  present  their  findings  to  "Item 
Team  Captains"  who  have  more  experience  and  understanding  of 
the  requirements  of  the  RFP.  The  Item  Team  Captain 
consolidates  the  factor  evaluations  to  determine  the  item 
rating  and  presents  the  evaluation  of  the  item  to  the 
"Area  Chief."  The  Area  Chief  then  rates  and  presents  the 
area  rating  to  the  "Source  Selection  Evaluation  Board 
(SSEB)."  The  SSEB  is  made  up  of  people  at  higher  level 
positions,  often  SPO  division  chiefs  (e.g.,  engineering, 
program  control,  logistics,  etc.),  and  is  chaired  by  the 
proqram  manager.  The  SSEB  preser ts  an  overall  evaluation  to 
the  "Source  Selection  Advisory  Council  <SSAC>."  On  "major" 
programs  (the  types  that  usually  include  C/SCSC) ,  the  SSAC 
normally  consists  of  General  Officers  or  members  of  the 
Senior  Executive  Service  (SES) .  The  SSAC  advises  the 
"Source  Selection  Authority  (SSA)  *'  who  determines  which 
contractor  will  be  awarded  the  contract.  The  SSA  is 
normally  the  Secretary  of  the  Air  Force  on  Air  Force 
proqrams  that  require  C/SCSC  (1:49-58).  Figure  1  presents 
Ihe  different  levels  in  the  source  selection  hierarchy 
including  what  each  level  evaluates. 

A  schedule  of  all  source  selection  activities  is 
developed  by  the  SPO  and  included  in  the  SSP.  This  schedule 
shows  the  time  allowed  to  complete  each  task  and  the  order 
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Figure  1.  Source  Selection  Hierarchy 
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in  which  the  events  need  to  be  started  and  completed 
(1:32-5).  Figure  2,  reprinted  -from  ASD's  The  Source 
Selection  Process ,  shows  an  example  of  a  Source  Selection 
Schedule  (1:214). 

The  Evaluation  Process.  The  ASD/PMPS  handbook 
explains  how  -factors  are  evaluated.  When  the  proposal 
satisfies  the  evaluation  standard,  a  "check"  is  recorded  in 
the  evaluation  form  for  the  factor.  If  the  proposal  exceeds 
the  requirements  of  the  standard,  a  '‘plus''  is  recorded. 

When  the  proposal  does  not  satisfy  the  standard,  the 
contractor  is  given  a  chance  to  correct  the  problem.  A 
"Deficiency  Report  (DR)"  notifies  the  contractor  that  the 
response  is  unacceptable.  If  the  contractor  does  not  change 
the  proposal  or  if  the  change  still  fails  to  satisfy  the 
standard,  a  "minus"  is  recorded  for  the  factor  (1:59-62). 

Helmig  explained  that  many  DRs  result  from  the 
contractor  misunderstanding  the  RFP.  These  problems  can 
easily  be  corrected  without  changing  the  basic  content  of 
the  proposal.  When  a  contractor  receives  a  DR,  only  the 
part  of  the  proposal  that  caused  the  problem  is  allowed  to 
be  changed.  The  DR  process  is  used  because  the  source 
selection  is  intended  to  choose  the  better  source  for  the 
acquisition,  not  to  eliminate  competitors  who  misinterpret 
the  RFP  (16) . 

A  "Detailed  Narrative  Analysis"  is  required  besides 
marking  a  "plus,"  "check,"  or  "minus"  for  the  factor.  This 


written  explanation  -for  the  rating  of  each  factor  should 
contain  a  description,  analysis,  summary,  and  rating.  The 
Detailed  Narrative  Analysis  reports  the  strengths  and 
weaknesses  along  with  the  risks  associated  with  the 
contractor's  approach  (1:99-101).  A  sample  of  a  Detailed 
Narrative  Analysis  is  included  in  Appendix  D. 

The  ASD/PMPS  handbook  also  explains  how  the  Item  Team 
Captain  should  indicate  the  rating  of  an  item  by  a  color 
code  after  all  the  factors  have  been  evaluated.  An 
outstanding  rating  should  be  coded  blue  while  an  acceptable 
rating  should  be  coded  green.  If  the  item  is  marginal  the 
rating  should  be  coded  yellow  whereas  an  unacceptable  rating 
receives  red.  The  reasons  for  the  rating  are  explained  in 
an  "Item  Summary"  which  is  similar  to  the  Detailed 
Narrative  Analysis  used  to  explain  the  reasons  for  a 
factor's  rating.  The  same  type  procedure  is  used  to 
determine  the  overall  rating  for  each  area  (1:61-2,  102—5). 
Samples  of  an  Item  Summary  and  an  "Area  Summary"  are  also 
included  in  Appendix  D. 

Daniel  Schild  mentioned  that  a  problem  in  rating  the 
items  and  areas  is  in  determining  the  effect  of  minuses  and 
pluses  on  the  total  rating  (i.e.  the  effect  of  a  "minus" 
when  everything  else  is  a  "plus").  It  is  for  this  reason 
that  the  lower-level  "write-ups"  must  contain  an  explicit 
statement  of  reasons  for  the  ratings  as  well  as  a  risk 
assessment  (20) . 

According  to  Schild,  only  critical  information  is 


included  in  the  SSEB '  s  presentation  o-f  the  area  ratings  to 
the  SSAC.  There  is  a  good  chance  that  problems  involving 
single— factor  "minus"  ratings  may  go  unnoticed  since  the  SSA 
and  SSAC  have  little  interface  with  the  factor  evaluators 
(20)  . 

A  final  important  fact  concerning  the  evaluation 
process  was  explained  by  James  Helmig.  The  source  selection 
process  does  not  require  all  areas  to  be  considered  equally. 
Normally,  the  Design  Approach  and  Life  Cycle  Cost  areas  are 
more  important  and  have  a  greater  impact  on  the  contract 
award  (16).  This  impact  of  contracts  containing  C/SCSC 
clauses  will  be  discussed  in  the  following  section. 

Research  Question  3. 

How  important  arm  the  C/SCSC-rmlatmd  areas  in  the 
overall  source  selection? 

C/SCSC  is  evaluated  at  the  factor  level.  In  an  example 
of  an  actual  major  program  presented  during  the  interview 
with  James  Helmig,  C/SCSC  was  located  under  the 
"Programming,  Management,  Planning,  and  Organization"  item 
in  the  "Manuf actur ing/Program  Management"  area.  Other 
factors  in  this  item  included  "Organizational 
Responsibilities,"  "Management  Information  System,"  “Work 
Breakdown  Structure,"  and  "Schedule"  (16). 

Figure  3  shows  the  relationship  between  C/SCSC  and 
other  factors,  items,  and  areas.  Many  of  the  possible  items 
and  factors  are  deleted  for  simplicity.  This  figure  does 
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Figure  3.  C/SCSC  in  Relation  to  Overall  Program 
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not  represent  any  particular  program  at  ASD.  As  Mr.  Helmig 
cautions,  this  illustration  represents  only  one  o-f  an 
endless  number  of  ways  that  a  Source  Selection  Plan  might  be 
designed.  A  SPO  can  add  and  delete  any  item  or  factor  as  a 
means  of  designing  an  SSP  to  satisfy  its  program 
requirements  (16). 

In  programs  at  ASD  where  C/SCSC  is  required,  ACCM  is 
the  sole  evaluator  of  the  C/SCSC  and  Work  Breakdown 
Structure  factors.  They  also  examine  the  Program  Control 
section  of  the  "Organizational  Responsibilities'*  factor. 

All  of  these  factors  are  normally  located  under  the  same 
item  (25).  There  is  a  difference  in  opinion  on  how  much 
influence  ACCM  has  in  the  rating  of  this  item. 

The  chief  of  ACCM,  Stan  Nesivich,  who  has  participated 
in  many  source  selections,  mentioned  that  the  evaluator  must 
prepare  an  accurate  "write-up"  regardless  of  the  evaluators 
influence  with  the  higher  levels  of  the  source  selection 
hierarchy.  In  order  to  strongly  influence  the  item  rating, 
ACCM  must  do  a  good  job  analyzing  the  strengths,  weaknesses, 
and  risks  of  the  contractor's  Comprehensive  Plan  for  C/SCSC. 
The  evaluator  must  also  be  sure  that  the  Item  Team  Captain 
understands  the  Detailed  Narrative  Analysis  of  each  factor 
(18)  . 

C/SCSC  has  limited  influence  in  determining  the 
rating  of  the  Manuf acturing/Program  Management  area.  C/SCSC 
is  only  one  of  approximately  50  factors  in  this  area.  One 


interviewee,  who  has  evaluated  many  contractors' 
Comprehensive  Plans  tor  C/SCSC  during  source  selection, 
doubts  that  a  ••minus"  on  C/SCSC,  when  all  other  factors  are 
either  a  "check"  or  "plus,"  would  result  in  a  "yellow"  or 
"red"  area  rating  <*) .  However,  another  interviewee  from 
the  same  office  believes  a  proposal  that  received  a  "minus" 
on  C/SCSC  would  likely  receive  "minuses”  on  other  factors  in 
the  Manuf acturing /Program  Management  area.  These  other 
"minuses"  could  result  in  a  "yellow"  or  "red"  rating  for  the 
area  (*)  . 

Mr.  Nesivich  mentioned  that  most  of  the  factors  in 
source  selection  receive  "green"  ratings.  Those  that  fail 
to  satisfy  the  standards  initially  are  often  acceptable 
after  the  DR  process.  There  is  no  rule  or  guideline  for 
determining  how  many  "red"  or  "yellow"  factors  are  needed  to 
rate  an  item  less  than  "green."  Therefore,  the  overall  area 
rating  is  based  on  the  SSEB ' s  judgment  of  how  well  the 
contractor  satisfies  the  area's  requirements  (18). 

Although  C/SCSC  is  only  a  small  part  of  the  overall 
proposal,  it  is  still  a  contract  requirement.  Dan  Schild 
recommends  when  the  contractor's  Comprehensive  Plan  for 
C/SCSC  is  bad  and  the  evaluator  feels  that  there  will  be 
difficulty  in  making  the  system  acceptable,  the  evaluator 
should  notify  the  Source  Selection  Evaluation  Board  about 
the  problem.  The  evaluator  can  even  suggest  that  the 
contractor  be  removed  from  competition,  although,  this  can 
be  overruled  by  others  on  the  SSEB  (20). 


What  specific  in-formation  should  ths  persons 
evaluating  Comprshsnsi vs  Plans  -for  C/8C8C  examine  in  the 
proposals? 

Preparing  -for  the  Evaluation.  A  proposal  received  by 
ASD  is  checked  to  ensure  all  required  items  are  included. 
William  Harper,  whose  job  includes  serving  on  source 
selections,  explains  that  on  RFPs  that  include  DOD  FAR 
Supplement  clause  52.242—7001,  it  is  ACCM's  responsibility 
to  ensure  that  the  proposal  includes  the  -following:  a 
Comprehensive  Plan  for  C/SCSC  (or  evidence  of  system 
validation),  a  Contract  Work  Breakdown  Structure  (broken 
down  to  the  cost  account  level),  a  CWBS  dictionary, 
identification  of  subcontractors  and  the  subcontracted 
effort,  and  procedures  for  the  flowdown  of  C/SCSC  to  major 
subcontractors.  The  contractor  is  notified  and  urged  to 
provide  the  information  if  any  of  these  items  are  not 
included  in  the  proposal  (15). 

As  mentioned  earlier,  ACCM  evaluates  the 
C/SCSC-related  areas  at  ASD.  Stan  Nesivich  explained  that 
normally  only  one  person  evaluates  the  proposals  for  a  given 
RFP.  The  people  that  evaluate  the  Comprehensive  Plans  for 
C/SCSC  are  the  more  experienced  members  of  his  staff.  These 
people  have  gained  C/SCSC  experience  from  serving  on  C/SCSC 
Demonstrati on/Val idation  Reviews  and  C/SCSC  Subsequent 
Application  Reviews.  This  experience  with  C/SCSC  reviews 
and  the  job  of  interpreting  contractor's  performance 


measurement  systems  helps  the  evaluator  make  "reasonable" 
decisions  about  the  acceptability  of  the  Comprehensive  Plan 
for  C/SCSC  (18). 

According  to  Wohlman,  it  normally  takes  3  to  5  days  to 
review  a  contractor's  Comprehensive  Plan  for  C/SCSC.  Part 
of  this  time  is  spent  waiting  on  clarif ications  of  parts  of 
the  contractor's  proposal.  Each  evaluation  normal  1 y  takes 
only  about  a  half-day  when  a  contractor  submits  evidence  of 
having  a  previously  accepted  system  (25). 

Limitations  and  Problems  with  the  Evaluation.  It  is 
important  to  understand  some  of  the  limitations  of  the 
source  selection  process  before  looking  at  the  specifics  of 
the  evaluation.  Mr.  Nesivich  mentioned  that  the  depth  of 
the  evaluation  in  the  Design  Approach  and  Life  Cycle  Cost 
areas  is  often  limited  by  the  time  allowed  to  review  and 
"write-up"  the  evaluation  of  each  proposal.  Time  is 
scheduled  for  the  evaluation  process  during  the  planning  for 
source  selection.  In  determining  how  much  time  to  spend  on 
a  proposal ,  the  reviewer  considers  the  total  amount  of  time 
available,  the  number  of  proposals  to  be  reviewed,  and  the 
number  of  reviewers.  Time  is  distributed  proportionately 
among  the  proposals  to  help  ensure  fairness  (18). 

Normally,  time  is  not  a  major  problem  in  evaluating  the 
C/SCSC  related  area  of  the  proposals.  According  to 
Nesivich,  the  "Design  Approach"  and  "Life  Cycle  Cost"  areas 
require  a  more  detailed  evaluation.  The  standards  for  these 


40 


areas  vary  greatly  from  program  to  program  and  the  people 
who  evaluate  these  factors  are  often  chosen  by  availability 
instead  of  ability.  Therefore,  the  evaluation  of  the 
factors  within  these  areas  often  takes  time.  The  C/SCSC  and 
WBS  standards,  however,  are  the  same  for  all  “major" 
programs.  The  stability  of  the  standards  and  the  experience 
of  the  evaluator  normally  result  in  a  timely  evaluation  of 
C/SCSC  (18).  Copies  of  ASD  evaluation  standards  for  C/SCSC 
and  Work  Breakdown  Structures  are  included  in  Appendix  C. 

Patrick  Cyrus,  an  ACCM  member  with  experience  in 
evaluating  Comprehensive  Plans  for  C/SCSC,  pointed  out 
another  problem  that  arises  when  the  government's  estimate 
for  the  program  cost  is  below  the  C/SCSC  threshold  but  the 
contractors'  proposals  exceed  the  C/SCSC  threshold.  This 
situation  fails  to  require  the  C/SCSC  clauses  in  the  RFP 
resulting  in  an  insufficiently  planned  evaluation  of  the 
C/SCSC.  In  this  case,  contractors  must  change  their 
proposal  to  include  the  Comprehensive  Plan  for  C/SCSC.  This 
either  shortens  the  evaluation  time  or  causes  delays  (7). 

Dan  Schild  adds  that  even  the  more  experienced 
evaluators  can  be  occasionally  fooled  by  a  contractor's 
Comprehensive  Plan  for  C/SCSC.  The  system  described  may 
adequately  satisfy  the  C/SCSC  requirements  but  be  totally 
different  than  the  system  the  contractor  plans  to  use.  This 
problem  can  be  identified  if  the  evaluator  has  time 
available  to  visit  the  plant  to  see  how  the  system  actually 


works  (20) 


The  Evaluation.  All  those  interviewed  expressed 
similar  views  concerning  how  to  evaluate  the  contractors' 
Comprehensive  Plans  for  C/SCSC.  This  section  summarizes  the 
findings  of  the  interviews. 

Eighty  percent  of  those  interviewed  stated  the 
evaluator  should  first  read  the  Comprehensive  Plan  for 
C/SCSC  from  "front  to  back"  to  ensure  the  plan  adheres  to 
the  general  philosophy  of  the  C/SCSC  concept.  Reading  the 
plan  helps  the  evaluator  determine  whether  the  plan  is 
noncontradictory  and  achievable  (15).  When  the 
Comprehensive  Plan  for  C/SCSC  is  not  understandable,  these 
evaluators  feel  there  is  little  reason  to  proceed  further  in 
the  evaluation  of  the  C/SCsC  factor  (7). 

All  of  the  interviewees  agreed  the  Comprehensive  Plan 
for  C/SCSC  must  show  how  the  contractor's  performance 
measurement  system  satisfies  the  C/SCSC  requirements. 

However ,  there  was  disagreement  on  how  much  detail  the  plan 
must  include.  The  responses  ranged  from:  (a)  requiring  the 
Comprehensive  Plan  for  C/SCSC  to  explain  how  every  one  of 
the  criteria  is  satisfied  <*)  to  (b)  simply  providing  enough 
explanation  to  demonstrate  that  the  contractor  understands 
the  35  C/SCSC  <•»)  .  Eighty  percent  of  those  interviewed 
agreed  the  most  important  criteria  disciplines  are  "earned 
value”  techniques,  "variance"  reporting,  material  costing 
techniques,  planning/work  package  procedures,  and  procedures 
for  applying  funds  to  planning/work  packages.  These 
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disciplines  are  where  major  problems  in  the  C/SCSC 
demonstration/  validation  process  often  occur  (25).  The 
remaining  20%  felt  that  all  criteria  disciplines  were 
equally  important.  One  of  those  interviewed  cautioned  that 
while  some  of  the  criteria  may  seem  more  important  in  source 
selection,  during  the  C/SCSC  Demonstrati on / Val idat ion  Review 
all  the  criteria  are  equally  important.  The  lack  of  any  of 
the  criteria  not  being  satisfied  by  the  contractor's  system 
is  sufficient  to  have  that  contractor's  entire  performance 
measurement  system  be  considered  unacceptable  (20) . 

All  those  interviewed  thought  the  35  criteria,  devoid 
of  the  subquestions  (the  bold  type  face  checklist  items)  of 
the  Joint  Implementation  Guide  Checklist,  are  a  useful  tool 
to  ensure  all  the  C/SCSC  are  discussed  by  the  contractor's 
Comprehensive  Plan  for  C/SCSC.  However,  all  the 
interviewees  explained  that  some  of  the  criteria  are 
difficult  to  evaluate  until  the  system  is  actually  in  use. 

As  one  evaluator  explained,  when  using  the  Joint 
Implementation  Guide  Checklist  as  a  guide,  the  evaluator 
needs  to  remember  that  during  source  selection  the  plan  need 
only  show  an  understanding  of  the  criteria.  Compliance  with 
the  C/SCSC  will  be  evaluated  after  the  contract  is  awarded 
(18)  . 

Sixty  percent  of  those  interviewed  agreed  that  most 
contractors  on  "major"  programs  have  some  understanding  of 
the  C/SCSC  requirements  from  responding  to  prior  RFPs. 

Hr.  Schild  explained  that  experienced  contractors  understand 
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the  government  evaluation  needs  and  format  their 
Comprehensive  Plan  for  C/SCSC  in  a  manner  that  is  easy  to 
evaluate.  Some  contractors  even  submit  a  completed  JIG 
checklist  as  a  cross-ref erence  between  each  criteria  and 
their  Comprehensive  Plan  for  C/SCSC  (20). 

All  of  the  evaluators  agreed  the  Comprehensive  Plan  for 
C/SCSC  either  satisfies  the  criteria  or  it  does  not.  This 
factor  should  not  receive  a  marginal  rating  in  their 
opinion.  Harper  explained  that  if  the  evaluator  has 
questions  or  concerns  with  the  plan,  he  should  contact  the 
contractor  to  get  the  needed  information  or  clarif ications 
(15)  . 

The  evaluators  unanimously  agreed  that  experience  is 
often  the  key  technique  in  the  evaluation.  In  many  cases, 
it  is  impossible  to  determine  whether  the  plan  will  perform 
as  proposed  until  the  system  is  in  use.  In  those  cases 
where  the  Comprehensive  Plan  for  C/SCSC  states  the  system 
will  work  in  a  certain  way,  the  evaluator  has  to  assume  that 
it  will  (15).  Therefore,  the  evaluation  is  often  based  on 
the  perception  that  the  contractor  understands  the  criteria. 
The  evaluator  must  rely  on  experience  to  make  this  type  of 
decision.  One  of  those  interviewed  mentioned  that  some 
evaluators  in  other  areas  are  often  timid  about  taking  a 
stand  on  an  issue,  however,  those  who  evaluate  C/SCSC  have 
the  experience  to  defend  the  decisions  they  make  (20). 

Eighty  percent  of  the  evaluators  mentioned  they  examine 
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the  contractor's  past  performance.  Many  of  the  contractors 
have  previously  done  some  type  of  work  for  the  government. 
These  evaluators  often  check  to  see  whether  the  contractor 
has  experience  with  the  "Contract  Fund  Status  Report  (CFSR) " 
or  the  "Cost /Schedule  Status  Report  (C/SSR)."  These  reports 
are  used  to  provide  the  government  funding  and  performance 
measurement  information  on  contracts  of  lesser  dollar  value. 
ASD,  as  well  as  other  procuring  organizations,  keeps  track 
of  perf ormance  measurement  on  past  and  present  programs  (7) . 

Sixty  percent  of  the  evaluators  explained  that  many 
contractors  mention  in  their  Comprehensive  Plans  for  C/SCSC 
the  hiring  of  experts  or  buying  of  computer  programs  to  help 
set-up  performance  measurement  systems.  Two/thirds  of  these 
individuals  do  not  consider  this  a  benefit  since  the  expert 
or  computer  program  may  not  smoothly  interface  with  the 
performance  measurement  system  currently  in  use.  According 
to  Harper,  the  evaluator  should,  whenever  possible,  look  at 
the  technical  description  of  the  acquisition  program  to  see 
whether  the  terms  used  are  similar.  This  will  demonstrate 
whether  the  people  who  designed  the  performance  measurement 
system  have  worked  together  with  the  technical  people  on  the 
project  to  design  a  compatible  system  <15). 

All  those  interviewed  agreed  that  when  the  contractor 
refers  to  a  previously  accepted  system,  the  evaluator  should 
verify  that  the  system  is  validated  and  currently  being 
used.  Mr.  Cyrus  explained  that  the  evaluator  should  also 
check  with  government  plant  representati ves  at  Contract 


Administration  Q-f-fices  and  with  other  government  sources  to 
determine  whether  the  validated  system  has  any  problems  or 
peculiarities.  These  common  problems  include  indirect 
costing  and  forward  pricing  agreements  <7) . 

Mr.  Harper  explained  that  during  source  selection,  one 
contractor  will  often  submit  an  outstanding  Comprehensive 
Plan  for  C/SCSC  while  another  contractor's  plan  will  only 
minimally  satisfy  the  standard.  Even  though  one  is  superior 
to  the  other,  both  would  normally  receive  a  satisfactory 
mark.  However,  the  evaluator  can  report  his  findings  in  the 
Detailed  Narrative  Analysis  (15). 

Summary  of  Research  Findings 

Interested  contractors  should  include  performance 
measurement  information  in  their  proposal  when  C/SCSC  is 
identified  as  a  contractual  requirement  of  a  solicitation. 
Contractors  should  include  a  Comprehensive  Plan  for  C/SCSC, 
a  CWBS  broken  down  to  the  cost  account  level,  a  CWBS  index, 
and  a  CWBS  dictionary.  A  contractor  currently  using  a 
C/SCSC  validated  system  may  submit  evidence  of  that 
validation  instead  of  the  Comprehensive  Plan  for  C/SCSC. 
Contractor's  also  need  to  identify  critical  subcontractors, 
their  subcontracted  effort,  and  procedures  for  C/SCSC- 
flowdown  to  these  subcontractors. 

C/SCSC  has  little  effect  in  the  overall  source 
selection  although  the  government  considers  it  very 
important.  Since  C/SCSC  is  evaluated  at  the  factor  level. 


which  is  normally  the  lowest  level  of  evaluation,  C/SCSC  can 
receive  a  unsatisfactory  rating  while  barely  influencing  the 
area  rating.  The  evaluator  can  make  strong  statements  in 
the  Detailed  Narrative  Analysis  in  order  to  influence  the 
Item  Team  Captain's  and  SSEB ' s  decision. 

The  DOD  does  not  have  a  standard  procedure  for 
evaluating  the  Comprehensive  Plans  for  C/SCSC  which  are 
submitted  by  contractors  in  response  to  RFPs.  Establishing 
guidelines  for  this  evaluation  is  a  process  normally 
delegated  to  the  procuring  agency.  AFSC  and  ASD  believe 
that  all  procuring  agencies  basically  use  the  same 
procedures,  although  this  research  effort  did  not  test  the 
validity  of  this  assumption. 

At  ASD,  the  Comprehensive  Plans  for  C/SCSC  are 
evaluated  by  experienced  members  of  the  Cost  Management 
Systems  Division  (ACCM) .  The  ACCM  evaluators  base  their 
evaluation  on  how  well  the  written  description  satisfies  the 
criteria  of  DODI  7000.2.  This  type  of  evaluation  is  based 
on  the  evaluator's  interpretation  of  the  criteria  and  the 
proposed  Comprehensive  Plan  for  C/SCSC.  The  evaluators  feel 
their  training  and  experience  in  C/SCSC  are  the  key  to 
performing  valid  evaluations. 

The  evaluators  explained  that  during  source  selection 
they  look  for  indications  that  the  contractors  have  a  good 
understanding  of  the  C/SCSC  concepts  rather  than  for  an 
absolute  compliance  with  the  criteria.  Trying  to  evaluate 


compliance  with  many  of  the  criteria  would  be  almost 
impossible  until  the  system  is  actually  in  use.  The  35 
criteria,  without  the  subquestions  contained  in  the  C/SCSC 
Joint  Implementation  Guide  Checklist,  are  a  good  guideline 
for  determining  what  areas  need  to  be  addressed.  When 
possible,  examining  the  past  performance  of  the  contractor's 
ability  to  report  cost  and  schedule  information  is  helpful 
in  determining  how  well  the  contractor  understands  the 
C/SCSC. 

As  one  of  the  interviewees  explained,  the  source 
selection  process  does  not  prevent  contractors  with  poor 
performance  measurement  systems  from  being  awarded  major 
contracts.  However,  it  does  alert  ACCIi  to  the  condition  of 
the  contractor's  performance  measurement  system. 


V. 


Conclusions 


The  objective  of  this  research  was  to  examine  how  the 
C/SCSC-related  areas  are  evaluated  during  source  selection. 
The  reason  for  researching  this  topic  was  that  occasionally 
a  contractor  with  an  inadequate  understanding  of  C/SCSC  is 
awarded  a  contract.  When  this  happens,  both  the  government 
and  the  contractor  have  inaccurate  and  untimely  performance 
measurement  information  to  use  in  managing  the  program. 

This  research  resulted  in  an  explanation  of  why  contractors 
with  inadequate  performance  measurement  systems  are 
sometimes  awarded  contracts. 

This  research  has  identified  two  major  problems  with 
the  current  approach  to  evaluating  C/SCSC  during  source 
selection.  They  ares  <1)  the  lack  of  emphasis  placed  on 
C/SCSC  and  <2)  the  fact  that  contractors  sometimes  provide 
false  information  in  the  Comprehensive  Plans  for  C/SCSC. 
Nevertheless,  C/SCSC  is  very  important  to  the  DOD  and  could 
grow  in  importance  as  more  emphasis  is  placed  on  controlling 
cost  and  reducing  schedule  delays.  However,  the  evaluators 
feel  that  C/SCSC  should  not  be  as  important  as  the  "Design 
Approach"  or  the  "Life  Cycle  Cost"  of  the  proposal. 

Despite  the  problems,  the  evaluators  at  ASD  seem 
satisfied  with  the  current  process.  None  of  the 
interviewees  off^ed  any  suggestions  for  improving  the 
source  selection  process.  Even  though  an  occasional 
contract  is  awarded  to  a  contractor  with  an  inadequate 


performance  measurement  system,  the  evaluation  still  helps 
ACCM  plan  the  activities  needed  to  validate  the  contractor's 
performance  measurement  system. 

The  product  of  this  research  was  stated  earlier  as  “a 
list  of  items  to  consider  when  evaluating  C/SCSC  during 
source  selection."  Only  a  few  highly  trained  and 
experienced  people  evaluate  the  C/SCSC-rel ated  areas  at  A6D 
during  source  selection.  Since  these  people  know  what  to 
look  for  in  this  process,  a  generic  description  of 
evaluation  would  have  limited  value  for  source  selection  in 
"major"  programs. 

However,  one  evaluator  at  ACCM  explained  that  a  listing 
of  important  items  to  consider  would  help  those  less- 
experienced  with  C/SCSC  understand  what  is  included  in  this 
contractual  requirement.  A  listing  of  important  C/SCSC 
characteristics  could  also  benefit  those  who  evaluate 
performance  measurement  on  programs  with  smaller  dollar 
val ue  (25) . 

The  following  section  identifies  items  to  consider  when 
evaluating  the  C/SCSC-rel ated  factors  during  source 
selection.  All  of  the  items  included  in  the  following  list 
are  discussed  earlier  in  this  thesis.  The  list  does  not 
include  any  procedures  or  guidelines  for  interpreting  the 
Comprehensive  Plans  for  C/SCSC  since  the  interviews  suggest 
that  experience  is  the  key  for  evaluating  the  C/SCSC— rel ated 
areas  during  source  selection. 
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C/SCSC  Items  to  Consider  During  Source  Selection 


1.  The  proposal  should  include: 

A.  Comprehensive  Plan  for  C/SCSC  or  proof 
of  C/SCSC  validation. 

B.  CUBS,  CUBS  Dictionary,  and  CWBS  Index. 

C.  Identification  of  critical  subcontractors 
and  their  subcontracted  effort. 

D.  Procedures  for  C/SCSC  flowdown  to 
subcontractors. 

2.  The  Comprehensive  Plan  for  C/SCSC  should: 

A.  Describe  how  the  contractor's  proposed 
systems  satisfies  the  C/SCSC  requirement. 

B.  Adhere  to  general  C/SCSC  philosophy. 

C.  Be  complete  and  non— contradictory. 

D.  Be  achievable. 

3.  The  Contract  Work  Breakdown  Structure  must: 

A.  Reflect  the  way  the  contractor  plans  to 
complete  the  program. 

B.  Be  an  extension  of  the  Project  Summary  WBS 
included  in  the  RFP. 

C.  Be  supported  by  a  CWBS  dictionary  and  a 
CWBS  i ndex . 

4.  If  the  contractor  submits  evidence  of  prior 

C/SCSC  validation,  check  to  ensure  that: 

A.  Contractor's  facility  is  still  operating 
under  the  system  that  was  validated. 

B.  Contractor  intends  to  use  the  same  system  if 
awarded  the  contract. 

5.  There  should  be  evidence  that  the  contractor 

is  using  or  intends  to  use  the  system  described 

in  the  Comprehensive  Plan  for  C/SCSC. 

6.  The  government  should  be  allowed  access  to  all 

information  and  supporting  documentation. 
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7.  If  the  contractor  has  performed  work  for  the 
DOD  before: 

A.  Check  with  the  ASD  or  other  data  base 
to  determine  if  the  contractor  provided 
accurate  and  timely  performance 
measurement  data. 

B.  Check  with  the  DCAA  Plant  Representati ve  to 
identify  any  peculiarities,  especially  with 
indirect  costing  and  forward  pricing 
methods. 

Based  on  the  interviews,  this  list  contains  the  key  points 
to  examine  during  source  selection.  The  evaluators 
emphasized  that  during  source  selection  an  understanding  of 
C/SCSC  is  expected  rather  than  a  demonstration  of  compliance 
with  the  criteria. 

Recommendation 

Related  research  may  be  beneficial  to  determine  how 
performance  measurement  systems  on  programs  with  smaller 
dollar  values  are  evaluated  during  source  selection.  For 
these  programs,  the  Cost /Schedule  Status  Report  (C/SSR)  and 
the  Contract  Funds  Status  Report  <CFSR>  are  required. 
According  to  Mr.  Wholman,  ACCM  does  not  normally  get 
involved  in  these  source  selections,  however,  ACCM  does 
review  their  RFPs  to  ensure  that  performance  measurement  is 
included.  These  type  contracts  are  often  evaluated  by  entry- 
level  people  of  the  SPO  who  lack  experience  in  performance 
measurement.  A  checklist  or  other  type  of  evaluation  tool 
could  be  beneficial  to  these  inexperienced  evaluators  (25). 
This  thesis  provides  information  that  could  be  useful  for 
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Appendix  A:  Evaluation/Demonstrat ion  Review  Checklist 
for  C/SCSC  Reprinted  from  (12:36-48) 


CHECKLIST  ITEMS 


I  ORGANIZATION 


I  DEFINE  ALL  THE  AUTHORIZED  WORK  AND  RELATED  RESOURCES  TO  MEET  THE  REQUIREMENTS  OF 
THE  CONTRACT.  USING  THE  FRAMEWORK  OF  THE  CWBS. 


a  Is  only  one  CWBS  used  for  the  contract  (attach  copy  of 
CWBS  i1 


b.  Is  all  contract  work  included  in  the  CWBS0 


c  Are  the  following  items  included  in  the  CWBS  (annotate 
copy  of  CWBS  to  show  elements  below)1 


(I )  Contract  bne  items  and  end  items  (if  in  consonance 
with  MIL  STD-881  A) 


(2)  All  CWBS  elements  specified  for  external  reporting. 


(3)  CWBS  elements  to  be  subcontracted,  with  identifica¬ 
tion  of  subcontractors 


(4)  Cost  account  levels. 

2  IDENTIFY  THE  INTERNAL  ORGANIZATIONAL  ELEMENTS  AND  THE  MAJOR  SUBCONTRACTORS  j 
RESPONSIBLE  FOR  ACCOMPLISHING  THE  AUTHORIZED  WORK.  1 


a  Are  ail  authorized  tasks  assigned  to  identified  organiza¬ 
tional  elements1  (This  must  occur  at  the  cost  account  level  as  a 
minimum.  Prepare  exhibit  showing  relationships.) 


b.  Is  subcontracted  work  defined  and  identified  to  the 
appropriate  u  be  on  tract  or  within  the  proper  WBS  element? 
(Provide  representative  example.) 


3.  PROVIDE  FOR  THE  INTEGRATION  OF  THE  CONTRACTOR  S  PLANNING.  SCHEDULING.  BUDGETING.  | 
WORK  AUTHORIZATION.  AND  COST  ACCUMULATION  SYSTEMS  WITH  EACH  OTHER.  THE  CWBS.  AND  THE  j 
ORGANIZATIONAL  STRUCTURE  (Reference  formal  I  ) 


a.  Are  the  contractor's  management  control  systems  listed 
above  integrated  with  each  other,  the  CWBS,  and  the  orgamza- 
nonaJ  structure  at  the  following  levels:  (Use  matrix  to  illustrate 
the  relationships.) 


(1)  Total  contract? 


(2)  Cost  account? 


4  IDENTIFY  THE  MANAGERIAL  POSITIONS  RESPONSIBLE  FOR  CONTROLLING  OVERHEAD  (INDIRECT 
COSTS).  | 


a  Are  the  following  organizational  elements  and  managers 
dearly  identified? 


(1)  Those  responsible  for  the  establishment  of  budgets 
and  assignment  of  resources  for  overhead  performance? 


AFSCP  173-5 


AFLCP  173-5 


DARCOM-P  715-5 


NAVMAT  P5240 


DLAH  8315.2 


(2)  Those  responsible  for  overhead  performance  control 
of  related  costs 


i  iii 

1  h  Are  the  responsibilities  and  authorities  of  each  of  the  I  | 

!  above  organizational  elements  or  managers  clearly  defined?  I 


!  5  PROVIDE  FOR  INTEGRATION  OF  THE  CWBS  WITH  THE  CONTRACTOR'S  FUNCTIONAL  ORGANIZA  j 

I  TIONAL  STRUCTURE  IN  A  MANNER  THAT  PERMITS  COST  AND  SCHEDULE  PERFORMANCE  MEASUREMENT  I 
FOR  CWBS  AND  ORGANIZATIONAL  ELEMENTS.  (Provide  matrix  showing  integration  ) 


b  Are  the  following  elements  for  measuring  performance 
available  at  the  levels  selected  for  control  and  analysis: 


!  HI  Budgeted  cost  for  work  scheduled0 


(21  Budgeted  cost  for  work  performed0 


(3)  Ac.ual  costs  of  work  performed0 


II  PLANNING  AND  BUDGETING 


,  1  SCHLDLTl  THE  AUTHORIZED  WORK  IN  A  MANNER  WHICH  DESCRIBES  THE  SEOUENCE  OF  WORK  AND 

i  IDENTIFIES  THF  SIGNIFICANT  TASK  INTER-DEPENDENCIES  REQUIRED  TO  MEET  THE  DEVELOPMENT. 
!  PRODUCTION.  AND  DELIVERY  REQUIREMENTS  OF  THE  CONTRACT. 


j  a  Does  the  scheduling  system  contain  (Prepare  exhibit 

|  showing  traceability  from  contract  task  level  to  work  package 
schedules.  1 


|  ( 1 1  A  master  program  schedule0 


C 2 1  Intermediate  schedules,  as  required,  which  provide  a 
lopcjl  sequence  from  the  master  schedule  to  the  cost  account 
level0 


(3)  Detailed  schedules  which  support  cost  account  and 
work  package  start  and  completion  dates/events0 


1  b.  Are  significant  decision  points,  constraints,  and  inter- 

j  faces  identified  r  .  key  milestones? 


i.  Does  the  scheduling  system  provide  for  the  identifies- 
■  non  of  work  progress  against  technical  and  other  milestones,  and 
I  also  provide  for  forecasts  of  completion  dates  of  scheduled  work? 


j  U  Arc  work  packages  formally  scheduled  in  terms  of 

physical  accomplishment  by  calendar  dates  (Gregorian,  Julian,  or 
manufacturing  da> 


2  IDENTIFY  PHYSICAL  PRODUCTS,  MILESTONES,  TECHNICAL  PERFORMANCE  GOALS,  OR  OTHER 
INDICATORS  THAT  WILL  BE  USED  TO  MEASURE  OUTPUT. 


AFSCP  !  73-5 


AFLCP  173-5 


DARCOM-P  715-5 


NAVMAT  P5240 
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Appendix  E  continued 


j  CHECKLIST  ITEMS  YES 


!  j  Are  meaningful  indicators  identified  for  use  in  measuring 

1  the  status  of  cost  and  schedule  performance0  (Provide  representa¬ 
tive  samples.! 


b  CK>es  the  contractor’s  system  identify  work  accomplish¬ 
ment  against  the  schedule  plan0  (Provide  representative  exam- 

,  pics  1 


.  Arc  current  work  performance  indicators  and  goals 
relaf.ible  to  original  goals  as  modified  by  contractual  changes, 
r 'planning,  and  reprogramming  actions0  (Provide  exhibit  showing 
ini  '(p> nation  ol  changes  to  original  indicators  and  goals  ) 


'  1  STABl  ISH  AND  MAINTAIN  A  TIME-PHASE  BUDGET  BASELINE  AT  THE  COST  ACCOUNT  LEVEL  AGAINST 
WHH’H  CONTRACT  PERFORMANCE  CAN  BE  “FASURED.  INITIAL  BUDGETS  ESTABLISHED  FOR  THIS  PURPOSE 
Will  BE  BASED  ON  THE  NEGOTIATED  TARGET  COST.  ANY  OTHER  AMOUNT  USED  FOR  PERFORMANCE 
Ml  \Sl  REMENT  PURPOSES  MUST  BE  FORMALLY  RECOGNIZED  BY  BOTH  THE  CONTRACTOR  AND  THE 
i«OVERNMLNT  (Reterence  formats  2  and  8.) 


i  Does  the  performance  measurement  baseline  consist  of 

i  he  VI I  owing0 


( l )  Time-phase  cost  account  budgets. 


(2)  Higher  level  CWBS  element  budgets  (where  not  yet 
broken  down  into  cost  account  budgets). 


(3)  Undistributed  budget,  if  any. 


(4)  Indirect  budgets,  if  not  included  in  the  above. 


b.  Is  the  entire  contract  planned  in  time-phased  cost 
accounts  to  the  extent  practicable0 


c.  In  the  event  that  future  contract  effort  cannot  be 
Jt'Wird  hi  sufficient  detail  to  allow  the  establishment  of  cost 
accounts,  is  the  remaining  budget  assigned  to  the  lowest 
practicable  CWBS  level  elements  for  subsequent  distribution  to 

- •  »n*  Jvcounts. 

d  Does  the  contractor  require  sufficient  detailed  planning 
ot  c.*t  accounts  to  constrain  the  application  of  budget  initially 
alto.\in\!  for  future  effort  to  current  effort0  (Explain  con- 

1  s{|  lints  | 


e.  Are  cost  accounts  opened  and  closed  based  on  the  start 
i ml  completion  ot  work  contained  therein0 


4  ESTABLISH  BUDGETS  FOR  ALL  AUTHORIZED  WORK  WITH  SEPARATE  IDENTIFICATION  OF  COST 
ELEMENTS  (LABOR.  MATERIAL,  ETC.).  (Reference  formats  2,  3,  and  4.) 


j.  Does  the  budgeting  system  contain  (Provide  exhibit.) 


( I )  The  total  budget  for  the  contract  (including 
estimates  for  authorized  but  unpriced  work)? 


AFSCP  173-5 


AFLCP  173-5 


DARCOM-P  715-5 


NAVMAT  P5240 


DLAH  8315.2 


Appendix  E -continued 


CHECKUST  ITEMS 


(2)  Budgets  assigned  to  major  fund  tonal  organizations'’ 
(See  checklist  Item  II.  9ab.) 


|  (3)  Budgets  assigned  to  cost  accounts0 


b  Are  the  budgets  assigned  to  cost  accounts  planned  and 
identified  in  terms  of  the  following  cost  elements?  (Reference 
Formats  3  and  4.) 


( 1 )  Direct  labor  dollars  and/or  hours. 


(2)  Material  and/or  subcontract  dollars. 


(3)  Other  direct  dollars. 


c.  Does  the  work  authorization  system  contain:  (Prepare 
sample  exhibit.) 


(1)  Authorization  to  proceed  with  all  authorized  work? 


(2)  Appropriate  work  authorization  documents  which 
subdivide  the  contractual  effort  and  responsibilities  within 
functional  organizations. 


5  TO  THE  EXTENT  THE  AUTHORIZED  WORK  CAN  BE  IDENTIFIED  IN  DISCRETE.  SHORT-SPAN  WORK 
PACKAGES.  ESTABI  l$H  Rl  IDGFTS  FOR  THIS  WORK  IN  TERMS  OF  DOLLARS.  HOURS.  OR  OTHER  MEASURABLE 
UNITS.  WHERE  THE  ENTIRE  COST  ACCOUNT  CANNOT  BE  SUBDIVIDED  INTO  DETAILED  WORK  PACKAGES. 
IDENTIFY  THE  FAR  TERM  EFFORT  IN  LARGER  PLANNING  PACKAGES  FOR  BUDGET  AND  SCHEDULING 
PURPOSES:  (Reference  formats  6. 6a.  and  6b.) 


a.  Do  work  packages  reflect  the  actual  way  in  which  the 
work  will  be  done  and  are  they  meaningful  products  or 
management-oriented  subdivisions  of  a  higher  level  element  of 
work?  (Provide  representative  sample.) 


b  Are  detailed  work  packages  planned  as  far  in  advance  as 
,  practicable0 


c.  Is  work  progressively  subdivided  into  detailed  work 
packages  as  requirements  are  defined? 


e  Are  work  packages  reasonably  short  in  time  duration  or 
do  they  have  adequate  objective  indicat ors/milcstoncs  to  mini¬ 
mize  the  in-process  work  evaluation0 


f.  Do  work  packages  consist  of  discrete  tasks  which  arc 
adequately  described0  (Provide  representative  sample.) 


g  Can  the  contractor  substantiate  work  package  and 
planning  package  budgets? 


h.  Are  budgets  or  values  assigned  to  work  packages  and 
planning  packages  in  terms  of  dollars,  hours,  or  other  measurable 
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CHECKLIST  ITEMS 

YES 

NO 

REMARKS 

i.  Are  work  packages  assigned  to  performing  organisations9 

t.  PROVIDE  THAT  THE  SUM  OF  ALL  WORK  PACKAGE  BUDGETS  PLUS  PLANNING  PACKAGES  WITHIN  A 
COST  ACCOUNT  EQUALS  THE  COST  ACCOUNT  BUDGET.  (Reference  formal  2.) 

a.  Does  the  sum  of  all  work  package  budgets  plus  planning 
packages  within  cost  accounts  equal  the  budgets  assigned  to  those 

cost  accounts9 

■ 

7  IDENTIFY  RELATIONSHIPS  OF  BUDGETS  OR  STANDARDS  IN  UNDERLYING  WORK  AUTHORIZATION 
SYSTEMS  TO  BUDGETS  FOR  WORK  PACKAGES. 


j  a  Where  engineered  standards  or  otheT  internal  work 

I  measurement  systems  are  used,  is  there  a  formal  relationship 
.  between  these  values  and  work  package  budgets?  (Provide 
|  samples  showing  relationships.) 


8  IDENTIFY  AND  CONTROL  LEVEL  OF  EFFORT  ACTIVITY  BY  TIME-PHASE  BUDGETS  ESTABLISHED  FOR 
THIS  PURPOSE  ONLY  THAT  EFFORT  WHICH  CANNOT  BE  IDENTIFIED  AS  DISCRETE.  SHORTSPAN  WORK 
I  PACKAGES  OR  AS  APPORTIONED  EFFORT  WILL  BE  CLASSED  AS  LOE.  (Reference  format  6.) 


a.  Are  time-phase  budgets  established  for  planning  and 
control  of  level  of  effort  activity  by  category  of  resource,  for 
example,  type  of  manpower  and/or  material9  (Explain  method  of 
control  and  analysis.) 


b  Is  work  properly  classified  as  measured  effort.  LOE.  or 
|  apportioned  effort  and  appropriately  separated9 

J  <1  ESTABLISH  OVERHEAD  BUDGETS  FOR  THE  TOTAL  COSTS  OF  EACH  SIGNIFICANT  ORGANIZATIONAL 
COMPONENT  WHOSE  EXPENSES  WILL  BECOME  INDIRECT  COSTS.  REFLECT  IN  THE  CONTRACT  BUDGETS  AT 
T1IF  APPROPRIATE  LEVEL,  THE  AMOUNTS  IN  OVERHEAD  POOLS  THAT  WILL  BE  ALLOCATED  TO  THE 
CONTRACT  AS  IN'DIRFCT  COSTS.  (Reference  DCAA  Audit  Manual  and  DAR  1 5  203.)  (Reference  format  7.) 


a  Are  overhead  cost  budgets  (or  projections)  established 

1  a  facility-wide  basts  at  least  annually  for  the  life  of  the 

..  i.ii  tract  * 

■ 

h.  Are  overhead  cost  budgets  established  for  each  organiza- 
i  non  which  has  authority  to  incur  overhead  costs? 

■ 

1  c  Are  all  elements  of  expense  identified  to  overhead  cost 

midgets  or  projections9 

■ 

d  Are  overhead  budgets  and  costs  being  handled  according 
|  t-  the  disclosure  statement  when  applicable,  or  otherwise 

1  properly  classified  (for  example,  engineering  overhead.  IRAD)? 

■ 

■ 

e  Is  the  anticipated  (firm  and  potential)  business  base 
projected  in  a  rational,  consistent  manner9  (Explain.) 

t.  Are  overhead  costs  budgets  established  on  a  basis 
.insistent  with  the  anticipated  direct  business  base? 

| 

■ 

i 

g  Are  the  requirements  for  ail  items  of  overhead  estab¬ 
lished  bv  rational,  traceable  processes9 

t 

■ 

■ 
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CHECKLIST  ITEMS 


h  Are  the  overhead  pools  formally  and  adequately 
identified?  (Provide  a  list  of  the  pools.) 


i  Are  the  organizations  and  items  of  cost  assigned  to  each 
pool  identified0 


j.  Are  projected  overhead  costs  in  each  pool  and  the 
associated  direct  costs  used  as  the  basis  for  establishing  interim 
rates  for  allocating  overhead  to  contracts? 


NAVMAT  PS 240 
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E 

k  Are  protected  overhead  rates  applied  to  the  contract 

r 

beyond  the  current  year  based  on- 

II)  Contractor  financial  periods;  for  example,  annual? 


12)  The  projected  business  base  for  each  period'* 


(3)  Contemplated  overhead  expenditure  for  each  penod 
based  on  the  best  information  currently  available0 


I  Are  overhead  protections  adjusted  in  a  timely  manner  to 
reflect  - 


( I )  Changes  in  the  current  direct  and  projected  base0 


(2)  Changes  in  the  nature  of  the  overhead  requirements0 


(3)  Changes  in  the  overhead  pool  and/or  organization 
structures0 


m  Are  the  WBS  and  organizational  levels  for  application  of 
the  projected  overhead  costs  identified? 


10  IDENTIFY  MANAGEMENT  RESERVES  AND  UNDISTRIBUTED  BUDGET. 


a.  Is  all  budget  available  as  management  reserve  identified 
and  excluded  from  the  performance  measurement  baseline0 


b.  Arc  records  maintained  to  show  how  management 
reserves  are  used0  (Provide  exhibit.) 


c.  Is  undistributed  budget  limited  to  contract  effort  which 
cannot  vet  be  planned  to  CWBS  elements  at  or  below  the  level 
specified  for  reporting  to  the  Government0 


'  .  d.  Are  records  maintained  to  show  how  undistributed 

^  |  budgets  arc  controlled0  (Provide  exhibit.) 

11  PROVIDE  THAT  THF  CONTRACT  TARGET  COST  PLUS  THF  ESTIMATED  COST  OF  AUTHORIZED  BUT 
UNPRICED  WORK  IS  RECONCILED  WITH  THE  SUM  OE  ALL  INTERNAL  CONTRACT  BUDGETS  AND 
MANAGEMENT  RESERVES.  (Reference  formats  3.  4.  and  5.) 


a  Does  the  contractor  s  system  description  or  procedures 
require  that  the  performance  measurement  baseline  plus  manage¬ 
ment  reserve  equal  the  contract  budget  base0 


AFSCP  173-5 


AFLCP  173-5 


DARCOM-P  715-5 


NAVMAT  P5240 


DLAH  8315.2 


Appendix  E -continued 


CHECKLIST  ITEMS 

YES 

NO 

REMARKS 

b.  Do  the  sum  of  the  cost  account  budgets  for  higher  level 
CWBS  elements,  undistributed  budget,  and  management  reserves 
reconcile  with  the  contract  target  cost  plus  the  estimated  cost  for 
authorized  unpriced  work’  (Provide  exhibit.) 

_ 

til.  ACCOUNTING 

1  RECORD  DIRECT  COSTS  ON  AN  APPLIED  OR  OTHER  ACCEPTABLE  BASIS  CONSISTENT  WITH  THE 
BUDGETS  IN  A  FORMAL  SYSTEM  THAT  IS  CONTROLLED  BY  THE  GENERAL  BOOKS  OF  ACCOUNT. 

a.  Does  the  accounting  system  provide  a  basis  for  auditing 
records  of  direct  costs  chargeable  to  the  contract0 

b.  Are  elements  of  direct  cost  (labor,  material,  and  so 
forth)  accumulated  within  cost  accounts  in  a  manner  consistent 
with  budgets  using  recognized  acceptable  costing  techniques  and 
controlled  by  the  general  hooks  of  account? 

2  SUMMARIZE  DIRECT  COSTS  FROM  THE  COST  ACCOUNTS  INTO  THE  WBS  WITHOUT  ALLOCATION  01  A 
SINGLE  COST  ACCOUNT  TO  TWO  OR  MORE  WBS  ELEMENTS.  (Reference  format  3.) 


j  a  Is  it  possible  to  summarize  direct  costs  from  the  cost 

|  account  level  through  the  CWBS  to  the  total  contract  level 
j  without  allocation  of  a  lower  level  CWBS  element  to  two  or  more 
I  higher  level  CWBS  elements?  (This  dt*s  not  preclude  the 
I  allocation  of  costs  from  a  cost  account  containing  common  items 
I  to  appropriate  using  cost  accounts.) 


3.  SUMMARIZE  DIRECT  COSTS  FROM  THE  COST  ACCOUNTS  INTO  THE  CONTRACTOR'S  FUNCTIONAL  \ 
ORGANIZATIONAL  ELEMENTS  WITHOUT  ALLOCATION  OF  A  SINGLE  COST  ACCOUNT  TO  TWO  OR  MORE  | 
ORGANIZATIONAL  ELEMENTS.  (Reference  format  4.) 


a.  Is  it  possible  to  summarize  direct  costs  from  the  cost 
account  level  to  the  highest  functional  organizational  level 
withoul  allocation  of  a  lower  level  organization's  cost  to  two  or 
more  higher  level  organizations’ 

4  RECORD  ALL  INDIRECT  COSTS  WHICH  WILL  BE  ALLOCATED  TO  THE  CONTRACT. 

a.  Does  the  cost  accumulation  system  provide  for  summari¬ 
zation  of  indirect  costs  from  the  point  of  allocation  to  the 
mn tract  total’ 

■ 

■ 

b  Are  indirect  costs  accumulated  for  comparison  with  the 
corresponding  budgets? 

i 

>•  Do  the  lines  of  authority  for  incurring  indirecl  costs 
.  •respond  to  the  lines  of  responsibility  for  management  control 
■!  the  same  components  of  costs’  (Explain  controls  for  fixed  and 
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[  CHECKLIST  ITEMS 


f.  Are  the  bases  and  rates  for  allocating  costs  from  each 
indirect  pool  to  commercial  work  consistent  with  those  used  to 
allocate  such  costs  to  Government  contracts9 


h.  Are  the  procedures  for  identifying  indirect  costs  to  j 

incurring  organizations,  indirect  cost  pools,  and  allocating  the  1 

costs  from  the  pools  to  the  contracts  formally  documented9  j 


5  IDENTIFY  THE  BASES  FOR  ALLOC  ATING  THE  COST  OF  APPORTIONED  EFFORT. 


a.  Is  effort  which  is  planned  and  controlled  in  direct 
relationship  to  cost  accounts  or  work  packages  identified  as 
apportioned  effort9 


b  Are  methods  used  for  applying  apportioned  effort  costs 
to  cost  accounts  applied  consistently  and  documented  in  an 
established  procedure9 


6  IDENTIFY  UNIT  COSTS.  EQUIVALENT  UNIT  COSTS.  OR  LOT  COSTS  AS  APPLICABLE. 


a.  Docs  the  contractor's  system  provide  unit  costs,  equiva¬ 
lent  unit  or  lot  costs  in  terms  of  labor,  material,  other  direct,  and 
indirect  costs?  (Describe  procedure.) 


b.  Does  the  contractor  have  procedures  which  permit 
identification  of  recurring  or  nonrecurring  costs  as  necessary? 


7  THE  CONTRACTOR'S  MATERIAL  ACCOUNTING  SYSTEM  WILL  PROVIDE  FOR.  ACCURATE  COST 
ACCUMULATION  AND  ASSIGNMENT  OF  COSTS  TO  COST  ACCOUNTS  IN  A  MANNER  CONSISTENT  WITH  THE 
BUDGETS  USING  RECOGNIZED,  ACCEPTABLE  COSTING  TECHNIQUES;  DETERMINATION  OF  PRICE  VARI¬ 
ANCES  BY  COMPARING  PLANNED  VERSUS  ACTUAL  COMMITMENTS;  COST  PERFORMANCE  MEASUREMENT  AT 
THE  POINT  IN  THL  MOST  SUITABLE  FOR  THE  CATEGORY  OF  MATERIAL  INVOLVED.  BUT  NO  EARLIER  THAN 
THE  TIME  OF  ACTUAL  RECEIPT  OF  MATERIAL;  DETERMINATION  OF  COST  VARIANCES  ATTRIBUTABLE  TO 
THE  EXCESS  USAGE  OF  MATERIAL.  DETERMINATION  OF  UNIT  OR  LOT  COSTS  WHEN  APPLICABLE;  AND  FULL 
ACCOUNTABILITY  FOR  ALL  MATERIAL  PURCHASED  FOR  THE  CONTRACT  INCLUDING  THF.  RESIDUAL 
INVENTORY. 
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CHECKLIST  ITEMS 

YES 

NO 

REMARKS 

e.  Does  the  contractor's  system  provide  for  the  determina¬ 
tion  of  cost  variances  attributable  to  the  excess  usage  of  material? 

J  f.  Does  the  contractor’s  system  provide  unit  or  lot  costs 

j  when  applicable? 

g.  Are  records  maintained  to  show  full  accountability  for 
all  material  purchased  for  the  contract,  including  the  residual 

!  inventory0 

i - 

I  IV  ANALYSIS 


1.  IDENTIFY  AT  THE  COST  ACCOUNT  LEVEL  ON  A  MONTHLY  BASIS  USING  DATA  FROM.  OR 
RECONCILABLE  WITH.  THE  ACCOUNTING  SYSTEM,  BCWS  AND  BCWP;  BCWP  AND  APPLIED  (ACTUAL  WHERE 
APPROPRIATE)  DIRECT  COSTS  FOR  THE  SAME  WORK;  VARIANCES  RESULTING  FROM  THE  ABOVE 
I  COMPARISONS  CLASSIFIED  IN  TERMS  OF  LABOR.  MATERIAL,  OR  OTHER  APPROPRIATE  ELEMENTS, 
I  TOGETHER  WITH  THE  REASONS  FOR  SIGNIFICANT  VARIANCES. 


a.  Does  the  contractor’s  system  include  procedures  for 
measuring  performance  of  the  lowest  level  organization  respon¬ 
sible  for  the  cost  account0  (Provide  typical  example.) 


b  Does  the  contractor's  system  include  procedures  for 
measuring  the  performance  of  critical  subcontractors? 


c.  Is  cost  and  schedule  performance  measurement  done  in  a 
consistent,  systematic  manner? 


d.  Are  the  actual  costs  used  for  variance  analysis  reconcil¬ 
able  with  data  from  the  accounting  system? 


e  Is  budgeted  cost  for  work  performed  calculated  in  a 
j  manner  consistent  with  the  way  work  is  planned?  (For  example, 
|  it  work  is  planned  on  a  measured  basis,  budgeted  cost  for  work 
I  performed  is  calculated  on  a  measured  basis.) 


f  Does  the  contractor  have  variance  analysis  procedures 
and  a  demonstrated  capability  for  identifying  (at  the  cost  account 
and  other  appropriate  levels)  cost  and  schedule  variances  resulting 
from  the  system  (provide  examples)  which  - 


(ll  Identify  and  isolate  problems  causing  unfavorable 

cost  variances'’ 


(2)  Fvaluate  the  impact  of  schedule  changes,  work- 

!  jiound,  etc'’ 


(3)  Evaluate  the  performance  of  operating  organiza- 

I  lions'1 

(4)  Identify  potential  or  actual  overruns  and  underruns? 


2.  IDENTIFY  ON  A  MONTHLY  BASIS.  IN  THE  DETAIL  NEEDED  BY  MANAGEMENT  FOR  EFFECTIVE 
!  CONTROL,  BUDGETED  INDIRECT  COSTS,  ACTUAL  INDIRECT  COSTS,  AND  VARIANCES,  ALONG  WITH  THE 
REASONS  (Reference  format  7.) 

i _ _ _ _ 
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CHECKLIST  ITEMS  YES 


a.  Are  variances  between  budgeted  and  actual  indirect  costs 
identified  and  analyzed  at  the  level  of  assigned  responsibility  for 
their  control  (indirect  pool,  department,  etc.)? 


Ib.  Does  the  contractor’s  cost  control  system  provide  for 
capability  to  identify  the  existence  and  causes  of  cost  variances 
resulting  from- 


( 1 )  Incurrence  of  actual  indirect  costs  in  excess  of 
budgets,  by  element  of  expense? 


(2)  Changes  in  the  direct  base  to  which  overhead  costs 
are  allocated? 


c.  Are  management  actions  taken  to  reduce  indirect  costs 
when  there  are  significant  adverse  variances? 


3.  SUMMARIZE  THE  DATA  ELEMENTS  AND  ASSOCIATED  VARIANCES  LISTED  IN  ITEMS  1  AND  2  ABOVE  ! 
THROUGH  THE  CONTRACTOR  ORGANIZATION  AND  WBS  TO  THE  REPORTING  LEVEL  SPECIFIED  IN  THF 
CONTRACT.  (Reference  formats  2,  3, 4,  5 ,  1 0,  and  I  1 .) 


a.  Are  data  elements  (BCWS.  BCWP,  and  ACWP)  progres¬ 
sively  summarized  from  the  detail  level  to  the  contract  level 
through  the  CWBS?  (Provide  exhibit.) 


b.  Are  data  elements  summarized  through  the  functional 
organizational  structure  for  progressively  higher  levels  of  manage¬ 
ment?  (Provide  exhibit.) 


c.  Are  data  elements  reconcilable  between  internal  sum¬ 
mary  reports  and  reports  forwarded  to  the  Government? 


d.  Are  procedures  for  variance  analysis  documented  and 
consistently  applied  at  the  cost  account  level  and  selected  WBS 
and  organizational  levels  at  least  monthly  as  a  routine  task? 
(Provide  examples.) 


4  IDENTIFY  ON  A  MONTHLY  BASIS  SIGNIFICANT  DIFFERENCES  BETWEEN  PLANNED  AND  ACTUAL  | 
SCHEDULE  ACCOMPLISHMENT  TOGETHER  WITH  THE  REASONS.  | 


(  a.  Does  the  scheduling  system  identify  in  a  timely  manner 

|  the  status  of  work?  (Provide  representative  examples.) 


!  b.  Does  the  contractor  use  objective  results,  design  reviews, 

|  and  tests  to  trace  schedule  performance?  (Provide  examples.) 


|  5  IDENTIFY  MANAGERIAL  ACTIONS  TAKEN  AS  A  RESULT  OF  CRITERIA  ITEMS  1  THROUGH  4  ABOVE. 


j  a  Is  data  disseminated  to  the  contractor's  managers  timely, 

•  accurate,  and  usable**  (Provide  examples.) 
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CHECKLIST  ITEMS 

YES 

NO 

REMARKS 

c.  Are  there  procedures  for  monitoring  action  items  and 
corrective  actions  to  the  point  of  resolution  and  are  these 
procedures  being  followed0 

_ 

6.  BASED  ON  PERFORMANCE  TO  DATE  AND  ON  ESTIMATES  OF  FUTURE  CONDITIONS,  DEVELOP  REVISED 
ESTIMATES  OF  COST  AT  COMPLETION  FOR  WBS  ELEMENTS  IDENTIFIED  IN  THF.  CONTRACT  AND  COMPARE 
THESE  WITH  THE  CONTRACT  BUDGET  BASE  AND  THE  LATEST  STATEMENT  OF  FUNDS  REQUIREMENTS 
REPORT  TO  THE  GOVERNMENT.  (Reference  formats  12.  13.  and  14.) 


a.  Are  estimates  of  costs  at  completion  based  on- 


( 1 )  Performance  to  date*’ 


(2)  Actual  costs  to  date? 


(3)  Knowledgeable  projections  of  future  performance9 


(4)  Estimates  of  the  cost  for  contract  work  remaining  to 
be  accomplished  considering  economic  escalation? 


h.  Are  the  overhead  rales  used  to  develop  the  contract  cost 
estimate  to  complete  based  on- 


(I )  Historical  experience0 


(2)  Contemplated  management  improvements0 


(3)  Projected  economic  escalation0 


(4)  The  anticipated  business  volume0 


c  Are  estimates  of  cost  at  completion  generated  with 
sufficient  frequency  to  provide  identification  of  future  cost 
problems  in  time  for  possible  corrective  or  preventive  actions  by 
both  the  contractor  and  the  Government  program  manager0 


d.  Are  estimates  developed  by  program  personnel  coordi¬ 
nated  with  those  responsible  for  overall  plant  management  to 
JiM ermine  whether  required  resources  will  be  available  according 
to  revised  planning0 


e.  Arc  estimates  of  cost  at  completion  generated  by 
knowledgeable  personnel  for  the  following  levels: 


( 1 )  Cost  accounts? 


(2)  Major  functional  areas  of  contract  effort? 


(3)  Major  subcontracts? 


(4)  WBS  elements  contractually  specified  for  reporting 
of  status  u>  the  Government  (lowest  level  only)? 


(5)  Total  contract  (all  authorized  work)0 


f.  Are  the  latest  revised  estimates  of  costs  at  completion 
compared  with  the  established  budgets  at  appropriate  levels  and 
causes  of  variances  identified0 
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CHECKLIST  ITEMS 

YES 

|  g  Are  estimates  of  cost  at  completion  generated  in  a 

I  rational,  consistent  manner?  Arc  procedures  established  for 
:  appropriate  aspects  of  generating  estimates  of  costs  at  comple- 

I  tion? 

»  h.  Are  estimates  of  costs  a*  completion  utilized  in 

determining  contract  funding  requirements  and  reporting  them  to 
the  Government? 


i  Are  the  contractors  estimates  ot  costs  at  completion 
reconcilable  with  cost  data  reported  to  the  Government9 


V  REVISIONS  AND  ACCESS  TO  DATA 


1.  INCORPORATE  CONTRACTUAL  CHANGES  IN  A  TIMELY  MANNER.  RECORDING  THE  EFFECTS  OF  SUCH 
CHANGES  IN  BUDGETS  AND  SCHEDULES.  IN  THE  DIRECTED  EFFORT  BEFORE  NEGOTIATION  OF  A  CHANGE. 
BASF  SUCH  REVISIONS  ON  THE  AMOUNT  ESTIMATED  AND  BUDGE  FED  TO  THE  FUNCTIONAL  ORGANIZA¬ 
TIONS. 


■  b.  Are  all  affected  work  authorizations,  budgeting,  and 

!  scheduling  documents  amended  to  properly  reflect  the  effects  of 
i  authorized  changes9  (Provide  examples.) 


j  c  Are  internal  budgets  for  authorized,  but  not  priced 

j  changes  based  on  the  contractor's  resource  plan  for  accompli  siring 
the  work? 


:  d  If  current  budgets  for  authorized  changes  do  not  sum  to 

'  the  negotiated  cost  for  the  changes,  does  the  contractor 
compensate  for  the  differences  by  revising  the  undistributed 
\  budgets,  management  reserves,  budgets  established  for  work  not 
j  yet  started,  or  by  a  combination  of  these? 


!  :  RECONCILE  ORIGINAL  BUDGETS  FOR  THOSE  ELEMENTS  OF  THE  WBS  IDENTIFIED  AS  PRICE  LINE 

ITEMS  IN  THE  CONTRACT.  AND  FOR  THOSE  ELEMENTS  AT  THE  LOWEST  LEVEL  OF  THE  DOD  PROJECT 
SUMMARY  WBS.  WITH  CURRENT  PERFORMANCE  MEASUREMENT  BUDGETS  IN  TERMS  OF  CHANCES  TO  THE 
!  AUTHORIZED  WORK  AND  INTERNAL  REPLANNING  IN  THE  DETAIL  NEEDED  BY  MANAGEMENT  FOR 
EFFECTIVE  CONTROL.  (Reference  formats  8  and  9.) 


j  Arc  current  budgets  resulting  from  changes  to  the 
authorized  work  and/or  internal  replanning,  reconcilable  to 
original  budgets  for  specified  reporting  items9 


3.  PROHIBIT  RETROACTIVE  CHANGES  TO  RECORDS  PERTAINING  TO  WORK  PERFORMED  THAT  WILL 
1  CHANGE  PREVIOUSLY  REPORTED  AMOUNTS  FOR  DIRECT  COSTS,  INDIRECT  COSTS,  OR  BUDGETS,  EXCEPT 
FOR  CORRECTION  OF  ERRORS  AND  ROUTINE  ACCOUNTING  ADJUSTMENTS. 


a  Are  retroactive  changes  to  direct  costs  and  indirect  costs 
prohibited  except  for  die  correction  of  errors  and  routine 
amounting  adiustmenu9 


h  Ate  direct  *n  indirect  cost  adjustments  being  accotn- 
j  plisfied  .k.-  'ding  to  accounting  procedures  acceptable  to  DC  A  A? 


AFSCP  173-5 


AFLCP  173-5 


DARCOM-P  715-5 


NAVMAT  PS 240 


DLAH  8315.2 


Appendix  E -continued 


CHECKLIST  ITEMS 

1 

t 

1 

c.  Are  retroactive  changes  to  BCWS  and  BCWP  prohibited 
except  for  correction  of  errors  or  for  normal  accounting 
adjustments? 

4.  PREVENT  REVISIONS  TO  THE  CONTRACT  BUDGET  BASE  EXCEPT  FOR  GOVERNMENT-DIRECTED 
CHANGES  TO  CONTRACTUAL  EFFORT. 


1 

i 

a.  Are  procedures  established  to  prevent  changes  to  the 
contract  budget  base  (see  definition)  other  than  those  authorized 
by  contractual  action? 

l 

» 

b.  Is  authorization  of  budgets  in  excess  of  the  contract 
budget  base  controlled  formally  and  done  with  the  full 
knowledge  and  recognition  of  the  procuring  activity?  Arc  the 
procedures  adequate? 

5.  DOCUMENT.  INTERNALLY,  CHANGES  TO  THE  PERFORMANCE  MEASUREMENT  BASEUNE  AND.  ON  A 
TIMELY  BASIS.  NOTIFY  THE  PROCURING  ACTIVITY  THROUGH  PRESCRIBED  PROCEDURES 


a.  Are  changes  to  the  performance  measurement  baseline 
made  as  a  result  of  contractual  redirection,  forma)  reprogram¬ 
ming.  internal  replanning,  application  of  undistributed  budget,  or 
the  use  of  management  reserve,  properly  documented  and 
reflected  in  the  Cost  Performance  Report? 

b.  Are  procedures  in  existence  that  restrict  changes  to 
budgets  for  open  work  packages,  and  are  these  procedures 
adhered  to’ 

i 

i 

c.  Are  retroactive  changes  to  budgets  for  completed  work 
specifically  prohibited  in  an  established  procedure,  and  is  this 
procedure  adhered  to’ 

d.  Are  procedures  in  existence  that  control  replanning  of 
unopened  work  packages,  and  are  these  procedures  adhered  to? 

6.  PROVIDE  THE  CONTRACTING  OFFICER  AND  DULY  AUTHORIZED  REPRESENTATIVES  ACCESS  TO  ALL 
OF  THE  FOREGOING  INFORMATION  AND  SUPPORTING  DOCUMENTS. 


a  Does  the  contractor  provide  access  to  all  pertinent  rec¬ 
ords  to  the  C/SCSC  Review  Team  and  surveillance  personneU 


Appendix  B:  Record  of.  Interviews 


This  appendix  summarizes  the  key  points  of  the 
interviews.  Some  areas  of  the  discussion  have  been  deleted 
(areas  referenced  by  a  *****  in  the  text  and  areas  without 
unanimous  agreement  in  the  evaluation  section)  in  order  to 
protect  privileged  information. 


INTERVIEW  1 

Lt  Col  Thomas  L.  Bowman  (5) 

AFIT/LSY  (Thesis  advisor) 

26  September  84  -  August  85 

PURPOSE 

To  discuss  the  progress  of  the  research  and  to  clarify 
information  given  by  others.  Lt  Col  Bowman  also  reviewed 
drafts  of  the  thesis  as  it  was  being  prepared.  His  changes 
and  comments  are  too  numerous  to  list.  Therefore,  only  the 
comments  that  are  referenced  in  the  text  are  included  in 
this  Record  of  Interview  (ROI). 

MAIN  POINTS 

Explained  need  for  the  research  done 
C/SCSC  regulations  since  1967 

Procurement  officials  lack  familiarity  with  C/SCSC 
Source  selection  members  have  difficulty  in  evaluation 
Regulations  not  clear  in  regards  to  evaluation 

In  reviewing  a  draft  explained  that: 

"a  WBS  Index  (which  is  a  matrix  of  which  WBS  elements 
compromise  the  Contract  Line  Items  and  End  Items)  is  also 
required.  The  WBS,  index,  and  dictionary  are  required  via  a 
single  'Data  Item  Descri pti on ' " 

Clarified  that  when  contractor  submits  Comprehensive  Plan 
for  C/SCSC,  the  contractor  is  certifying  that  his  system 
satisfies  the  criteria. 

Clarified  that  the  contractor's  performance  measurement 
system  is  not  proven  or  demonstrated  as  adhering  to  the 
C/SCSC  concept  until  after  contact  award. 


INTERVIEW  2 


Patrick  L.  Cyrus  <7> 

Cost  Management  Systems  Division  (ASD/ACCM) 
1  May  1985 


PURPOSE 

Pat  has  evaluated  C/SCSC  during  several  source 
selections.  The  interview  was  to  determine  how  he  evaluates 
the  C/SCSC  area,  and  any  other  comments  he  has  about  the 
C/SCSC  evaluation  process  during  source  selection. 


EVALUATION  KEYS 

Experience  and  understanding  of  C/SCSC  is  most  important. 

A  lot  is  learned  from  past  experiences. 

Read  the  plan  to  make  sure  it  makes  sense,  little  value  in 
preceding  if  it  does  not. 

Plan  must  show  how  contractors  system  will  satisfy  C/SCSC. 

The  plan  should  provide  enough  detail  to  show  that 
contractor  understands  the  C/SCSC  concept. 

THE  JIG  checklist  identifies  what  the  plan  needs  to  address; 
however,  determining  if  the  plan  adequately  discusses  each 
criteria  is  difficult. 

Important  areas  are  earned  value,  material,  breaking  program 
down,  and  applying  funds  to  work. 

The  evaluator  does  not  rate  each  criteria  separately,  either 
the  whole  system  is  acceptable  or  none  of  it. 

If  the  contractor  claims  use  of  current  approved  system, 
always  verify. 


OTHER  COMMENTS 


On  some  occasions  the  estimated  cost  is  below  C/SCSC 
thresholds,  but  the  proposals  exceed  the  threshold.  As  a 
result  the  RFP  must  be  modified  to  include  C/SCSC  clauses. 


Check  ASD  and  other  procuring  organizations  -for  records  of 
contractors'  past  performance.  If  contractor  has  experience 
with  CFSR  or  C/SSR  check  quality  of  information  provided. 

Look  to  see  if  contractor  has  hired  consultants  or  is  using 
computer  programs  in  establishing  performance  measurement 
system.  If  he  has  it  will  be  mentioned  early  in  the  plan. 

Check  with  plant  representati ve  to  find  out  if  the 
contractor  has  any  peculiarities  or  problems,  especially 
with  indirect  cost.  In  some  case  the  the  contractor  has 
disputed  areas  such  as  forward  pricing  agreements. 

COMr^NTS  ON  LIST 

Most  important  items  are  included  in  the  list,  but  need  to 
expand  on  importance  of  the  criteria  listed  in  the  JIG 
checkl ist . 


INTERVIEW  3 


William  R.  Harper  <15) 

Cost  Management  Systems  Division  (ASD/ACCM) 
9  May  1985 


PURPOSE 

Bill  has  also  evaluated  C/SCSC  in  several  source  selections. 
He  was  asked  to  explain  how  C/SCSC  is  evaluated  during 
source  selection  and  -for  any  additional  in-formation  on 
C/SCSC. 


EVALUATION  KEYS 

Look  to  see  whether  system  description  or  evidence  o-f 
validation,  CWBS  (to  account  level), CWBS  dictionary,  list  o-f 
subcontractors  and  their  effort,  and  procedures  for  C/SCSC 
flowdown  are  included  in  proposal.  If  not  included, 
contractor  is  asked  to  provide  items. 

Completely  read  the  comprehensive  description  to  see  that 
contractor  has  understanding  of  the  C/SCSC  requirements, 
satisfy  the  criteria. 

The  evaluator  should  determine  if  the  whole  system  satisfies 
the  C/SCSC  requirement.  However,  remember  that  a  validated 
system  is  not  required  until  program  is  blooming. 

Variance  reporting,  material  costing,  indirect  costing, 
budgeting,  and  breakdown  of  tasks  are  areas  where  problems 
often  arise.  There  should  be  procedures  to  describe  these 
areas. 

Practical  experience  is  essential  in  interpreting  the 
contractor's  proposed  performance  measurement  system. 

The  main  items  of  the  JIG  checklist  are  a  good  outline  to 
ensure  that  all  important  areas  are  discussed. 


OTHER  COMMENTS 

Check  the  files  at  ASD  to  verify  the  validation  of  a 
contractors  system  and  also  look  for  other  past  performance 
information  that  may  be  useful  in  the  evaluation. 

Programs  under  C/S  thresholds  may  require  C/S  if  required  by 
government  and  stipulated  in  RFP. 


The  cost  -for  contractor  to  establish  validated  C/SCSC  system 
is  approximately  SOOK. 

It  the  evaluator  has  questions  or  concerns  with  the  proposed 
system,  he  should  check  with  the  contractor  to  get 
clarifying  information. 

If  the  contractor  states  the  system  will  work  in  a  specified 
way,  the  evaluator  has  to  assume  it  will,  unless  he  has 
information  to  prove  the  contrary. 

When  possible,  tie  technical  reports  in  with  the  evaluation 
of  the  Comprehensive  Plan  for  C/SCSC.  This  shows  whether 
performance  measurement  people  are  working  with  the 
technical  side  of  the  program. 

Do  not  get  impressed  by  contractors  mentioning  the  use  of 
experts  or  computers  in  their  description.  Each  system  is 
unique,  therefore,  this  may  not  be  a  benefit. 

Evaluation  determines  if  system  is  acceptable,  not  which 
contractor  has  best  system.  Even  though  other  bidders  have 
better  performance  measurement  systems,  each  contractor 
should  be  rated  on  own  merits  only.  However,  be  sure  to 
explain  the  risk  involved  in  each  contractor's  write-up. 


COMMENTS  ON  LIST 

Thought  all  the  items  listed  are  important  in  the 
evaluation,  but  suggested  rewriting  several  items. 

Would  be  interested  in  looking  at  list,  but  for  list  to  have 
impact  it  should  be  directed  and  distributed  by  ACC. 


INTERVIEW  4 


James  R.  Helmig  (16) 

Chief,  Source  Selection  Division  (ASD/PMPS) 

9  May  19S5 

PURPOSE 

Jim  is  very  experienced  in  the  source  selection  process.  He 
was  asked  to  explain  the  source  selection  process  and  how 
C/SCSC  relates  to  the  overall  source  selection.  Many  of  the 
areas  discussed  are  explained  in  the  ASD/PMPS  handbook. 
Therefore,  these  areas  are  not  included  in  this  ROI. 


KEY  POINTS 
Program  Makeup 

Area  (  oper  utility,  readiness  support,  life  cycle  cost, 
design  approach,  mnfg/program  mgmt) 

Items  -  8-10  per  area 

Factors  -  several  for  each  item 
Subfactors  —  seldom  used 

The  program  is  broken  down  into  smaller  parts  to  allow  for 
easier  and  accurate  evaluation. 

There  is  almost  an  endless  number  of  items  and  factors  that 
can  be  used  (provided  a  list  of  common  names). 

All  the  areas,  items,  and  factors  are  included  in  the 
program's  unique  source  selection  plan.  The  program  should 
not  copy  the  SSP  or  use  the  same  items  and  factors  as 
another  program.  The  SPO  can  use  any  items  and  factors  it 
chooses  and  place  any  value  on  the  factors,  but  the  SSP  must 
be  approved. 

The  DR  process  often  results  from  contractors 
misunderstanding  RFP.  The  contractor  is  allowed  to  change 
only  the  bad  area  of  proposal.  This  Process  is  used  because 
some  good  contractors  could  be  eliminated  because  of  minor 
errors  without  it. 

The  design  approach  is  normally  the  most  important,  LCC  is 
also  important  in  many  source  selections.  Management,  where 
C/SCSC  is  located,  is  often  ranked  low  in  order  of 
i mportance. 

In  a  example  of  a  recent  major  program,  C/SCSC  was  located 
in  "Programming ,  Management,  Planning,  and  Organization" 
item  along  with  the  "Organizational  Responsibilities", 
"Management  Information  Systems",  "WBS" ,  and  "Schedule" 
factors. 


INTERVIEW  5 


Stanley  J.  Nesivich  (18) 

Chief,  Cost  Management  Systems  Division  (ASD/ACCM) 
14  May  1985 


PURPOSE 

Stan  evaluated  proposals  before  becoming  chief.  He  was 
asked  to  explain  the  evaluation  process  and  provide  any 
other  comments  on  how  C/SCSC  is  evaluated  during  source 
selection. 


KEY  POINTS 

Make  sure  the  plan  makes  sense  before  preceding  with 
detailed  evaluation.  The  plan  helps  the  government 
determine  if  the  contractor  understands  the  C/SCSC  concept. 

Ideally  look  at  all  criteria,  but  often  cannot  fully 
evaluate  to  system  is  in  use.  The  big  type  items  in  the  JIG 
checklist  are  a  good  guide  for  determining  what  is 
important. 

Key  criteria  areas  arei  earned  value  techniques,  variance 
reporting,  and  material  costing. 

Evaluator  experience  is  key  in  successful  evaluations. 

Before  working  on  source  selection,  the  evaluator 
participates  in  several  C/SCSC  reviews. 

If  any  part  of  the  contractor's  plan  cannot  be  modified  to 
be  acceptable  than  the  whole  plan  is  unacceptable. 

Get  proof  that  contractor  is  actually  using  a  validated 
system,  when  it  is  claimed. 


OTHER  COMMENTS 

Normally  only  person  works  a  source  selection.  Because  the 
evaluators  are  all  highly  trained  and  experienced,  they  are 
able  to  make  reasonable  decisions  about  the  acceptability  of 
the  proposal 

Must  have  time  to  adequately  review  and  write-up  all 
evaluations.  Limitation  on  review  caused  by  time,  number  of 
proposal ,  and  number  of  evaluators.  Time  is  distributed 
proportionately  to  help  ensure  fairness. 


Normally,  time  is  not  a  problem  because  the  design  and  cost 
areas  are  more  difficult  to  evaluate.  Their  standards 
differ  with  every  program  and  the  evaluators  are  often 
inexperienced,  while  the  C/SCSC  area  is  relatively  stable. 

Source  selection  only  determines  an  understanding  of  C/SCSC. 
Actual  compliance  is  not  required  until  three  months  or 
longer  after  contract  award. 

Most  factors  receive  green  ratings,  especially  after  the  OR 
process.  There  is  no  rule  for  determining  how  many  red  or 
yellow  factors  are  needed  to  rate  an  item  or  area  less  than 
green. 

The  evaluator  must  explain  the  evaluation  carefully  in  the 
write-up  to  help  the  higher  levels  evaluate. 

Management  panel  has  less  pull  than  cost  and  technical 
panels. 

Hiring  of  consultants  or  experts  is  not  necessarily  a 
benefit  since  they  may  not  be  familiar  with  contractor's 
existing  performance  measurement  system.  Computer  programs 
are  not  necessarily  a  benefit  either. 

ESD,  SD,  !<  AD  evaluate  C/SCSC  basically  the  same  way. 


COMMENTS  ON  LIST 

Liked  the  areas  in  the  list,  but  suggested  either  discussing 
the  criteria  as  a  whole  or  discussing  every  criteria,  not 
just  key  criteria. 

Even  though  the  ACCM  people  are  experienced,  the  list  could 
benefit  them  and  would  definitely  help  inexperienced  people 
see  what  is  included  in  C/SCSC. 


INTERVIEW  6 


Donald  A.  Osborne  (19) 

Chief,  Resources  Validation  Division  (A8D/ACCC) 

27  March  1985 

PURPOSE 

To  determine  if  ACCC  has  any  responsibilities  in  regards  to 
C/SCSC ,  and  to  discuss  how  Life  Cycle  Cost  is  evaluated. 

KEY  POINTS 

ACCC  does  not  get  involved  in  C/SCSC. 

Life  Cycle  Cost  area  evaluated  by  ACCC  and  SPO. 

SPO  people  often  chosen  for  source  selection  by  availability 
instead  of  experience. 

Approximately  half  of  the  evaluators  in  a  cost  panel  have 
source  selection  experience. 

Standard  instruction  for  LCC  in  RFP. 

LCC  area  should  be  realistic  and  complete. 

LCC  also  determines  O&S  cost  based  on  design. 

LCC  summarized  by  most  probable  cost. 


INTERVIEW  7 


Daniel  Schild  (20> 
C/SCSC  Consultant,  TASC 
15  May  1985 


PURPOSE 

Dan  was  former  chief  of  ACCM,  he  new  consults  with  ASD.  He 
was  asked  to  describe  how  he  would  evaluate  C/SCSC  during 
source  selection  and  for  any  other  comments  on  C/SCSC. 


KEY  POINTS 

System  needs  to  minimally  satisfy  each  of  the  35  criteria. 

All  the  criteria  are  equally  important. 

Smart  contractor  will  include  filled  out  JIG  checklist  cross- 
referenced  with  the  proposal  to  help  the  evaluator. 

Need  experience  and  training  to  properly  evaluate. 

Best  way  to  evaluate  is  to  use  bold  face  items  of  checklist. 

First  read  proposal  to  get  familiar  with  contractors 
performance  measurement  system. 

System  either  satisfies  factor  or  it  does  not.  The 
write-up  should  explain  reasons  for  rating. 

The  evaluator  should  verify  that  the  contractor  is  using  an 
validated  system. 


OTHER  COMMENTS 

Some  criteria  may  seem  more  important  during  source 
selection  but  in  the  validation  review  any  of  the  criteria 
can  cause  the  system  to  be  unacceptable. 

When  system  is  bad  and  contractor  refuses  to  change  to  meet 
requirements,  the  evaluator  should  make  sure  that  others 
know  of  problem.  Can  attempt  to  have  contractor  removed 
from  competition,  but  this  can  be  overruled  by  others. 

Evaluator  may  be  timid  about  giving  a  contractor  a  bad 
review  and  making  a  big  deal  about  it  but  experience  helps 
in  making  judgments. 
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Check  with  other  DOD  procuring  organizations  -for  past 
performance  information  on  contractor. 

FAR  clause  7000.2  provides  assurance  that  contractor  intends 
to  use  the  performance  measurement  system  described  in 
proposal . 

The  source  selection  does  not  always  identify  potential 
problems  with  the  performance  measurement  system. 

Even  experienced  evaluators  can  be  fooled.  Some  contractors 
submit  comprehensive  plans  wrote  by  other  organizations, 
without  adequate  planning  on  how  to  integrate  C/SCSC  system 
with  existing  system. 

When  time  permits,  visit  facility  to  see  how  system  actually 
works. 

Hard  to  determine  effect  of  bad  C/SCSC  rating  on  items, 
areas,  and  overall  evaluation.  Therefore,  the  write-ups, 
that  include  risk  assessment,  are  very  important. 

Lack  of  interface  between  evaluators  and  SSAC  causes  single 
factors  to  sometimes  go  unnoticed.  Normally  only  critical 
summary  information  is  presented  to  higher  levels  of  source 
selection  management. 

By  evaluating  C/SCSC  during  source  selection,  the  government 
reduces  the  probability  that  a  contractor  without  proper 
performance  measurement  procedures  will  be  awarded  a 
contract. 


COMMENTS  ON  LIST 

Thought  the  list  would  be  useful  for  those  without  C/SCSC 
background. 


INTERVIEW  8 


Lt  Col  Connie  O.  Teetz  (22) 

Chie-f,  Cost  Information  Systems  Division  (AFSC/ACCI) 
21  March  1985 


PURPOSE 

Lt  Col  Teetz 's  division  is  the  Air  Force's  focal  point  for 
C/SCSC.  He  was  asked  questions  dealing  with  how  the  Air 
Force  evaluates  C/SCSC  during  source  selection. 


KEY  POINTS 

There  is  not  an  Air  Force  focal  point  in  charge  of  source 
selection.  Each  procuring  division  has  the  freedom  to 
establish  source  selection  guidelines  as  long  as  they  are 
consistent  with  the  regulations. 

ASD  and  ESD  are  the  only  two  product  divisions  that  have 
full  time  source  selection  people,  however,  these  people  are 
involved  in  organization,  noL  the  selection  itself. 

Different  products  divisions  can  require  different 
information  to  examine  non-val idated  systems,  but  the  C/SCSC 
requirements  must  be  consistent  with  the  FAR  and  AFR  70-15. 

Proposal  must  reference  approved  system  or  explain  general 
procedures  for  performance  measurement. 

There  is  no  AFSC  guideline  for  evaluating  C/SCSC  during 
source  selection. 

AFSC  keeps  up  with  what  the  product  divisions  do,  and  there 
is  little  difference  in  the  way  that  C/SCSC  is  evaluated 
during  source  selection. 

Validated  systems  do  not  receive  an  advantage  over  non- 
val  idated  systems  in  the  source  selection. 


SPECIAL  COMMENT 

David  Graham  and  Ron  Phillips  of  the  same  office  (AFSC/ACCI) 
as  Lt  Col  Teetz  were  also  interviewed  by  telephone.  All 
three  explained  the  same  information.  Since  Lt  Col  Teetz  is 
the  division  chief,  he  is  referenced  in  the  text. 


INTERVIEW  9 


John  A.  Wohlman  (25) 

Cost  Management  Systems  Divisions  (ASD/ACCM) 
18  April  1985 


PURPOSE 

John  has  evaluated  the  C/SCSC  related  areas  on  several 
source  selections.  He  was  asked  to  explain  how  he  evaluates 
the  C/SCSC  related  areas  and  -for  other  comments  that  deal 
with  C/SCSC  during  source  selection. 


KEY  POINTS 

Always  independently  veri-fy  that  contractor  is  using  a 
validated  system  when  it  is  claimed. 

Experience  is  needed  to  make  judgments  in  determining  i-f 
minimum  requirements  are  met. 

The  plan  either  satis-fies  the  standard  or  it  does  not,  i-f 
their  is  a  fuzzy  area  talk  with  contractor. 

Contractors  need  to  show  understanding  of  C/SCSC,  not  full 
compliance  with  the  criteria. 

JIG  checklist  is  a  nice  guideline,  but  very  difficult  to 
evaluate  several  of  the  items. 

Key  items  are  those  that  frequently  cause  problems  during 
demonstration  reviews,  some  of  these  are  earned  value, 
material  costing,  overhead,  breaking  down  the  work,  and 
applying  funds  to  the  work  package. 


OTHER  COMMENTS 

ASD  has  large  data  bank  of  contractors  past  performance  with 
regards  to  performance  measurement. 

Most  contractors  have  bid  on  DOD  programs  before. 

Therefore,  they  know  how  to  structure  the  response. 

Smaller  companies  have  difficulty  understanding  the  process 
and  often  have  to  adjust  their  proposal. 

ACCM  evaluates  C/SCSC,  WBS,  and  Program  Control  section  of 
organizational  responsibilities  factor  during  source 
sel ect i on. 


The  standards  are  the  same  -for  all  programs  (provided  copy 
of  standards) . 

All  RFPs  are  brought  through  ACCM  for  review  prior  to 
release,  to  ensure  that  proper  performance  measurement 
clauses  are  included.  Though  SPO  can  design  RFP  to  meet  the 
needs  of  program,  but  regulations  must  be  followed.  On 
programs  that  include  C/SCSC,  FAR  clause  7000. 1  is  included. 

The  contractor  must  submit  a  comprehensive  description  of 
the  performance  measurement  system  or  evidence  of 
validation,  MBS  and  MBS  dictionary,  and  information  on 
subcontractors  and  subcontracted  effort.  A  summary  MBS  is 
given  to  the  contractor,  who  must  extend  the  MBS  to  cost 
account  level. 

Even  though  comprehensive  description  of  performance 
measurement  is  acceptable  in  source  selection,  it  does  not 
mean  system  will  be  acceptable  to  a  validation  team. 

An  initially  unacceptable  response  to  a  factor  can  be 
corrected  via  the  DR  process  or  face-to-face  interview 
before  the  final  rating. 

All  ACCM  people  are  experienced  enough  to  evaluate  in  source 
selection.  Normally  only  one  person  looks  at  this  section 
of  proposal . 

A  validated  system  can  be  evaluated  in  a  half-day,  while  a 
comprehensive  description  takes  three  to  five  days,  but  this 
includes  time  to  clarify  responses. 


COMMENTS  ON  LIST 

A  checklist  would  be  useful  for  those  who  evaluate 
performance  measurement  on  smaller  programs. 

He  liked  the  items  in  the  checklist,  but  made  a  few  minor 
corrections  and  suggested  that  the  MBS  area  be  expanded. 


APPENDIX  C:  ASP  Evaluation  Standards 
(reprinted  -from  information  provided  by  John  Wohlman  (25)) 

ITEM:  Management  M.3 

FACTOR:  Program  Control  M.3. 10 

DESCRIPTION:  This  item  considers  the  definitive 

plans  for  the  offerors  application  of  management  tools  and 

techniques  required  by  the  S.O.W. 

The  offerors  planned  implementation  and  utilization  of 
a  Cost /Schedule  Management  System  will  be  evaluated  to 
assess  the  capability  to  plan  cost  and  schedule  elements  and 
to  measure  performance  against  these  elements.  This  system 
to  be  capable  of  demonstrating  compliance  with  DODI  7000.2 
and  meets  the  criteria  of  AFSCP  173-5. 

The  CUBS  proposed  by  the  offeror  will  expand  the  PWBS 
to  the  lowest  level  and  include  all  work  on  the  contract. 
Offerors'  CWBS  will  satisfy  the  intent  of  MIL-STD-B81A. 


ITEM:  Management  M.3 


FACTOR:  Cost  Schedule  Control  5ystems  M.3. 10.1 

DESCRIPTION:  This  item  considers  the  contractors'  proposed 

Cost  Management  System  designed  to  measure  contract 
performance  in  accordance  with  DODI  7000.2,  Performance 
Measurement  For  Selected  Acquisitions  and  meet  the  criteria 
in  AFSCP  173-5. 

STANDARD:  The  standard  is  met  when: 

Complete  documentation  is  provided  pertaining  to  the 
system  for  accumulation  of  data  to  prepare  and  submit  cost 
performance  reports.  The  documentation  should  be  in  the 
form  of  a  comprehensive  system  description  applicable  to  R&D 
or  production  efforts  and  compliant  with  DODI  7000.2. 

If  the  bidder  proposes  to  operate  under  a  D0D  validated 
system  for  either  or  both  R&D  and  production,  the  validation 
of  the  system  may  be  established  by  providing  a  copy  of  the 
validation  letter  or  memorandum  of  understanding  to  that 
effect. 

Identify  major  subcontractors  or  subcontracted  effort 
planned  for  application  of  the  criteria.  Describe  the 
proposed  procedure  for  administration  of  the  criteria  as 
applied  to  subcontractors. 


ITEM:  Management  M.3 


FACTOR:  Work  Breakdown  Structure  M.3. 10.2 

DESCRIPTION:  This  considers  the  expansion  of  the  PWBS  to 

the  lowest  level  CWBS.  The  CWBS  should  comply  with  the 
procedures  and  policies  contained  in  MIL-STD-881A. 

STANDARD:  The  standard  is  met  when: 

The  CWBS  and  dictionary  is  based  upon  and  is  an 
extension  of  the  PWBS  and  satisfies  the  intent  of  MIL-STD- 
881  A. 

The  CWBS  and  dictionary  satisfactorily  expands  the 

PWBS. 

The  CWBS  represents  all  costs  in  the  contract. 

All  subcontracted  elements  are  identified  in  the  CWBS. 
Elements  of  the  CWBS  are  clearly  defined  so  that 
duplication  of  effort  within  and  between  organizations  will 


not  occur . 


Appendix  D:  Evaluation  Write-ups 
Reprinted  f rom  (1:99-105 ) 


FOR 

OFFICIAL  USE  ONLY 

|  DETAILED  NARRATIVE  ANALYSIS  | 

1  SOURCE  SELECTION  EVALUATION  BOARD 

(i) 

»TCM 

(2) 

EA-  TOR 

(3) 

SUB* ACTO« 

(4) 

COMPANV 

(5) 

(6) 

I  . 

DESCRIPTION.  (Brief  description  of  what  th* 
proposed  -  Include  operational  impact,  if  applicable.) 

II  . 

ANALYSIS. 

A.  Strengths  ) 

) 

) 

) 

) 

B.  Weaknesses  ) 

) 

) 

) 

) 

C.  Risks  ) 

These  paragraphs  should  address  the 
following: 

Relevant  Past  Performance 
Understanding  of  the  Problem 

Soundness  of  Proposed  Approach 
Compliance  with  Requirements 
Suitability  of  Best  Commercial 
Practices,  if  applicable 
Deficiencies  and  Correction 

Potential 

(V) 

Ill  . 

SUMMARY . 

( Scrubbed 

down  Paragraph  II). 

IV. 

RATING. 

(+  Exceeds  Minimum) 

(v  Meets  Minimum) 

(-  Below  Minimum) 

(8) 

DETAILED  NARRATIVE  ANALYSIS  (PREPARATION  OF) 


PURPOSE:  This  (oris  is  used  to  -furnish  the  SSAC  and  the 

SSA  a  description  o-f  what  the  company  proposed  in 
a  satisfaction  o-f  an  element  o-f  the  evaluation  and 
to  report  the  extent  that  the  company's  submission 
meets  the  standard  -for  that  element.  While  it 
must  contain  enough  information  to  present  a  clear 
picture  to  the  review  authorities,  it  must  not 
contain  detailed  description  of  the  company's  data 
or  aircraft,  nor  is  it  necessary  to  delve  into  the 
extremely  detailed  analysis  in  the  justification 
of  your  findings.  In  other  words,  be  complete  but 
concise.  Fill  in  each  block  in  accordance  with 
the  following  instructions. 


(1)  Indicate  the  name  and  short  title,  if  assigned,  of 

the  SSEB  or  SSEC. 

Example:  Long  Range  Turbo-Prop 

Aircraft  (MX) 


(2)  Indicate  the  numerical  and  descriptive  designator 

of  item  involved. 

Example:  Item  H  Logistics  Support 

Item  A  Systems  Analysis 


(3)  Indicate  the  factor  numerical  and  descriptive 

designator . 

Example:  Factor  B. 1  Range 

Factor  H.3  Logistics  Analysis 


(4)  If  sub  -factors  have  been  identified  as  a  shred— out 

of  the  factor,  and  standards  have  been  prepared 
for  them  at  the  sub-factor  level,  indicate  the 
numerical  and  descriptive  designator. 

Example:  Sub-factor  T. 1.1  Master  Plan 

B. 1 . 3  Ferry 


(5)  Indicate  the  company's  name. 


(6)  Include  the  narrative  analysis  of  the  element  for 

that  company.  Preface  each  paragraph  of  the 
narrative  with  the  proper  security  classification 
and  downgrading  group. 


Indicate  the  symbolic  and  adjectival  designator  in 
parenthesis,  immediately  -following  the  end  of  the 
past  paragraph  of  the  narrative  analysis. 

Ex  amp  1  e  : 

.  .  .  .  were  presented  in  proper  order. 
(Meets  Minimum  ). 


Ex ample: 


.  .  .  .  although  it  affects  the 

acceptability,  it  can  be  easily  corrected. 
(Below  Minimum  -). 


Place  the  highest  overall  classification  at  the 
top  and  bottom  of  the  sheet.  Include  the 
statement  "FOR  OFFICIAL  USE  ONLY"  at  the  top  and 
bottom  of  the  page  notwithstanding  other  security 
classifications. 


AD-A161  649  EVALUATION  OF  PROCEDURES  EHPLOVED  DURING  SOURCE 

SELECTION  FOR  CONTRACTS  I.  .  <U>  AIR  FORCE  INST  OF  TECH 
NRIGHT-PATTERSON  AFB  OH  SCHOOL  OF  SVST.  .  J  E  PUGH 
UNCLASSIFIED  SEP  05  AFIT/GSH/LSV/85S-M  FFG  5/1 


ITEM  SUMMARY  (PREPARATION  OF) 


PURPOSE: 


(1) 


(2) 


<3> 


(4) 


(5) 


(6) 


The  Item  Summary,  which  is  used  in  the  Summary 
Report,  must  furnish  the  SSAC  ant  the  SSA  a 
concise  summation  of  the  strong  and  weak  points  of 
the  company's  proposal  which  was  evaluated.  Fill 
in  each  block  in  accordance  with  the  following 
i nstructions. 


Indicate  the  name  and  short  title  (if  assigned)  of 
the  SSEB  or  SSEC. 

Example:  Long  Range  Turbo-Prop  Aircraft  (MX) 


Indicate  the  designators  for  the  item. 
Example:  L.l  Logistics  Support 


Indicate  the  name  of  the  company. 
Example:  The  XXX  Company 


Narrative  of  the  summarized  findings.  Include  the 
security  classification  and  downgrading  grouping 
for  each  paragraph. 

Type  the  adjective  rating  and  numerical  score 
assigned  (in  parenthesis)  immediately  following 
the  end  of  the  last  paragraph  of  the  item  summary. 
Example:  ...  were  presented  in  proper  sequence. 

(Meets  Minimum  —  5) 

(Below  Minimum  -  4) 


Indicate  the  highest  security  classification  at 
top  and  bottom  of  the  page.  Include  the  statement 
"FOR  OFFICIAL  USE  ONLY"  at  top  and  bottom  of  the 
page  notwithstanding  other  security 
classification. 


_ FOK  OFFICIAL  USI  ONLY 

area  summary 

SOU*C£  SC  LCC  *'ON  CVAUUATIOH  aOARO 

(1)  (2) 


COMPANY 


(3) 


(«) 


A  SO 


PO«*  1(0 

0«C  M 


rot  OFFICIAL  use  OMLY 


89 


%*  "  ' 

V_V;/ 


W.-V 


AREA  SUMMARY  (PREPARATION  OF) 


PURPOSE: 


(1) 


(2) 


(3) 


<  4 ) 


The  area  summary  document  is  prepared  -for  each 
area  by  the  Area  Chief  with  assistance  of  team 
captains.  The  summary  contains  a  brief  resume  of 
the  findings  of  the  teams  of  the  area  and  includes 
statements  pertaining  to  the  interaction  af  these 
item  findings  to  the  area  evaluation.  Normally, 
the  summary  briefly  states  how  the  company 
approaches  the  solution  to  the  problem  considered 
to  be  most  critical  to  the  area  and  how  well,  in 
the  judgment  of  the  Area  Chief,  this  solution 
solves  the  problem.  Major  outstanding  features 
should  be  cited  as  well  as  major  deficiencies  and 
weaknesses  and  high  risk  approaches.  Because  of 
the  type  of  information  contained  in  these 
summaries,  they  become  a  prime  tool  in  the 
selection  process.  As  such,  they  require  care  in 
preparation  and  intensive  coordination.  Fill  in 
each  block  in  accordance  with  the  following 
instructions. 


Indicate  the  name  and  short  title  (if  assigned)  of 
the  SSEB  or  SSEC. 

Example:  Long  Range  Turbo-Prop 

Aircraft  (MX) 


Insert  the  numerical  and  descriptive  designator 
for  the  area. 

Example:  T.O  Technical  Area 

L.O  Logistics  Area 
C.O  Cost  to  the  Government  Area 


Indicate  the  name  of  the  company. 
Example:  The  XXX  Company. 


In  narrative  style,  indicate  a  summary  of 
findings.  Preface  each  paragraph  with  the 
security  classification  of  the  page  at  the  top  and 
bottom  of  the  sheet.  Include  the  statement  "FOR 
OFFICIAL  USE  ONLY”  at  top  and  bottom  of  the  page 
notwithstanding  other  security  classifications. 
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